From 4affbd98e97f24cbd7f51349faddbacaf81b80ac Mon Sep 17 00:00:00 2001 From: Franklin Moormann Date: Sun, 9 Aug 2026 10:03:47 -0400 Subject: [PATCH 1/9] docs: preregister cross-client mistake transfer --- .../CROSS_CLIENT_MISTAKE_TRANSFER_PROTOCOL.md | 237 ++++++++++++++++++ 1 file changed, 237 insertions(+) create mode 100644 docs/CROSS_CLIENT_MISTAKE_TRANSFER_PROTOCOL.md diff --git a/docs/CROSS_CLIENT_MISTAKE_TRANSFER_PROTOCOL.md b/docs/CROSS_CLIENT_MISTAKE_TRANSFER_PROTOCOL.md new file mode 100644 index 000000000..55f9d12bc --- /dev/null +++ b/docs/CROSS_CLIENT_MISTAKE_TRANSFER_PROTOCOL.md @@ -0,0 +1,237 @@ +# Cross-client mistake-transfer protocol + +Status: pre-registered implementation plan +Base commit: `59656c2c64399165157ad2ba6564fef5a0991798` +Primary live matrix: Codex `gpt-5.6-sol` and Claude Code `claude-sonnet-5` + +## Question + +Can one active coding model encounter a mistake, correct it, record the durable +lesson through `wiki_write`, and cause a fresh agent in another CLI to avoid the +same preventable mistake while completing related work? + +The existing evidence does not answer this question. A seeded finding proves +delivery, and a single-client A/B run can estimate retrieval overhead, but +neither proves natural semantic capture or behavioral transfer between agents. + +## Claims kept separate + +1. **Graph integrity:** concurrent and sequential writers do not corrupt or lose + findings. +2. **Natural capture:** the producer records the target lesson without a seeded + finding or detached semantic model. +3. **Cross-client delivery:** the consumer receives that exact finding with + producer/client/model provenance. +4. **Preventive timing:** delivery occurs before execution of the preventable + action, not merely after its failure. +5. **Behavioral transfer:** the consumer avoids the producer's mistake and still + completes the task correctly. +6. **Efficiency:** any behavioral gain is worth its token, call, latency, and + monetary cost. + +No lower-numbered claim is evidence for a higher-numbered claim. Deterministic +fixtures may establish integrity and grading, but only live model runs count as +behavioral evidence. + +## Experimental unit + +One matched pair begins with exactly one fresh producer run. After the producer +finishes, the harness freezes: + +- the post-producer workspace; +- the natural graph and its hash; +- a redacted producer ledger; +- the producer client/model/session identity; and +- hidden grader state recording the mistake, recovery, and final result. + +The harness then clones the same post-producer workspace for every consumer +arm. Producer transcripts, CLI session files, memory files, debug logs, and raw +responses are removed before cloning. The graph is the only producer-derived +artifact that differs between consumer arms. + +A producer that fails to make, correct, or harvest the target lesson remains in +the dataset as an end-to-end capture failure. It is never silently excluded. + +## Consumer arms + +| Arm | Integration | Graph contents | Purpose | +| --- | --- | --- | --- | +| `empty` | full local hooks/MCP | empty | isolates ordinary client behavior with equal optimizer overhead | +| `natural` | full local hooks/MCP | producer's active-model graph | measures the complete product | +| `oracle` | full local hooks/MCP | only a pre-registered ideal finding | estimates the upper bound of perfect harvesting | +| `irrelevant` | full local hooks/MCP | applicable-looking but unrelated finding | measures retrieval precision and distraction | +| `stale` | full local hooks/MCP | invalidated target finding | measures stale-evidence harm avoidance | + +An optional `no-system` arm may estimate total plugin/MCP overhead, but it is not +the primary graph-transfer contrast. The primary causal comparison is +`natural` versus `empty` because both load the same integration. + +Consumer order uses a generalized cyclic Latin rotation. Every arm occupies +every order position before the rotation repeats. Each consumer is a fresh CLI +process with a fresh session and hook-state directory. + +## Scenarios + +### Verification entry point + +The producer is required to try a tempting unsupported verification script, +observe the failure, recover through the package-level command, and complete a +small change. The consumer receives a related subsystem whose tempting script +has the same failure mode. + +Hidden audit files distinguish an attempted command from an executed command +and from a successful supported verification. Final prose alone cannot pass. + +### Generated source of truth + +The producer first changes a generated client copy, observes the synchronization +failure, then corrects the source module and regenerates the clients. The +consumer must make an analogous change for another generated client. + +The hidden grader checks which file was changed, whether generated outputs are +in sync, and whether the behavioral test passes. + +### False-positive validation + +The producer runs a plausible validation from the wrong scope, observes that it +did not exercise the target, and corrects it with the scoped command. The +consumer must validate a sibling target where the same false-positive trap is +present. + +The hidden grader requires the target sentinel to be exercised. A zero exit +code from the wrong command is recorded as a preventable mistake, not success. + +### Negative and stale controls + +Irrelevant guidance must not be delivered or followed. Guidance whose anchor or +invalidator changed after capture must be withheld, marked stale, or explicitly +re-verified before use. + +## Natural-finding contract + +The producer finding must be `origin: agent` and include: + +- a concrete claim describing the corrected failure; +- a real anchor; +- observed evidence; +- applicability; +- `verified`, `probable`, or `speculative` confidence; +- project/organization/global scope; and +- at least one invalidator. + +The target-finding grader uses pre-registered semantic anchors and observable +behavior. It does not require one exact sentence. A weak, unrelated, or missing +finding is recorded as a capture failure even if another finding exists. + +## Pre-registered outcomes + +### Primary + +- `attemptedPreventableMistake`: the consumer selected the known bad action. +- `executedPreventableMistake`: the bad action actually ran. +- `mistakeRecurrenceRate`: consumers with the executed mistake divided by all + included consumers in the arm. +- `correct`: hidden behavioral grader passes and the runner completes. +- `firstPassSuccess`: the first relevant validation exercises the target and + succeeds. + +Attempted and executed mistakes are separate because a preventive hook may +correctly block an action after the model selects it but before execution. + +### Transfer chain + +- `captureSuccess` and semantic-contract completeness; +- producer client/model/session and finding IDs; +- `findingDelivered`, delivery surface, and delivery order relative to the first + relevant tool call; +- `findingAdheredTo`: the next relevant behavior follows the finding; +- exact/fallback causal join coverage; and +- graph and redacted-artifact hashes. + +### Efficiency and harm + +- uncached, cache-creation, cached, output, and total tokens; +- tool calls, failed tool calls, latency, and cost; +- injected tokens and time-to-correct completion; +- irrelevant/stale delivery rate; +- new errors attributable to transferred guidance; and +- harmful feedback/quarantine events. + +Positive efficiency deltas always mean the treatment improved the metric. +Correctness and mistake prevention take precedence over token savings. + +## Isolation and exclusions + +Both runner profiles must declare exact baseline isolation and exact local +treatment configuration. The harness verifies loaded client, version, model, +plugin, MCP server, graph directory, and session identity before accepting a +cohort. + +Allowed exclusions are limited to: + +- authentication or provider outage before model work begins; +- runner/fixture failure before model work begins; and +- machine termination that leaves no gradable artifact. + +Timeouts, missing findings, incorrect work, malformed findings, repeated +mistakes, and missing token usage remain recorded. Missing values stay `null` +and are never converted to zero. + +No `AGENTS.md`, `CLAUDE.md`, auto-memory, transcript, prior CLI session, or git +commit message may carry the producer lesson into a consumer arm. All arms +receive the same code and git state. Raw transcripts are excluded by default; +published ledgers retain identities, grades, whitelisted tool classes, and +cryptographic hashes. + +## Live matrix + +The first effectiveness claim requires both directions: + +1. Codex / `gpt-5.6-sol` producer to Claude Code / `claude-sonnet-5` consumer. +2. Claude Code / `claude-sonnet-5` producer to Codex / `gpt-5.6-sol` consumer. + +Same-client fresh-agent cohorts (Codex to Codex and Claude to Claude) separate +cross-session behavior from CLI interoperability. A mixed concurrent cohort +runs two producers against one graph and a third fresh consumer after both +finish. + +Packaged lifecycle simulations still cover every supported client, but they are +reported as wiring certification rather than live behavioral proof. + +## Claim gates + +Five matched pairs per direction are the minimum dashboard display threshold. +Ten matched pairs per direction are required before making an effectiveness +claim for a scenario. + +For a positive cross-client mistake-transfer claim: + +- target natural capture succeeds in at least 80% of producer runs; +- the natural arm reduces executed mistake recurrence by at least 50% versus + empty graph; +- the paired 95% interval for recurrence improvement excludes zero; +- natural-arm correctness is not more than 10 percentage points below empty; +- delivery is observed before execution in at least 80% of successful captures; +- the irrelevant and stale controls introduce no correctness regression; and +- the complete redacted ledger, including failures, is published. + +The concurrent cohort additionally requires a parseable graph, zero lost +accepted writes, unique producer provenance, and successful retrieval of every +applicable finding in all repetitions. + +If a gate fails, the result is published as mixed, harmful, or not established. +Zero observed mistakes in a finite cohort is evidence for that cohort, never a +guarantee that an agent will make no future mistakes. + +## Required implementation and evidence + +- [ ] Generic sequential producer/consumer runner. +- [ ] Reproducible hidden-grader fixtures for all three mistake families. +- [ ] Natural, empty, oracle, irrelevant, and stale graph arms. +- [ ] Pre-action delivery available consistently across lifecycle clients. +- [ ] Cross-client episode/finding/delivery/outcome provenance. +- [ ] Concurrent live-writer and later-reader runner. +- [ ] Transfer and recurrence aggregation in the evidence API/dashboard. +- [ ] Deterministic protocol, isolation, security, and UI tests. +- [ ] Live bidirectional, same-client, negative-control, and concurrent cohorts. +- [ ] Redacted result reports and an honest PR evidence summary. From 2bd93043b5a047d4f4117e991b790ef1db71d515 Mon Sep 17 00:00:00 2001 From: Franklin Moormann Date: Sun, 9 Aug 2026 10:40:16 -0400 Subject: [PATCH 2/9] test: measure cross-client mistake transfer --- evals/README.md | 30 + .../mistake-transfer/clients/alpha/policy.txt | 1 + .../mistake-transfer/clients/beta/policy.txt | 1 + evals/fixtures/mistake-transfer/package.json | 11 + .../mistake-transfer/scripts/audit.mjs | 9 + .../mistake-transfer/scripts/check-alpha.mjs | 4 + .../mistake-transfer/scripts/check-beta.mjs | 4 + .../mistake-transfer/scripts/sync.mjs | 20 + .../mistake-transfer/scripts/test.mjs | 18 + .../mistake-transfer/scripts/validate.mjs | 17 + .../mistake-transfer/scripts/verify-alpha.mjs | 5 + .../mistake-transfer/scripts/verify-beta.mjs | 5 + .../mistake-transfer/source/alpha-policy.txt | 1 + .../mistake-transfer/source/beta-policy.txt | 1 + .../fixtures/mistake-transfer/src/alpha.json | 3 + evals/fixtures/mistake-transfer/src/beta.json | 3 + .../mistake-transfer/targets/alpha.json | 3 + .../mistake-transfer/targets/beta.json | 3 + evals/handoff-suite.json | 124 ++++ hooks-core/adapter.mjs | 27 +- hooks-core/inject.mjs | 80 ++- hooks-core/metrics.mjs | 171 ++++- .../hooks/token-optimizer/lib/adapter.mjs | 27 +- .../hooks/token-optimizer/lib/inject.mjs | 80 ++- .../hooks/token-optimizer/lib/metrics.mjs | 171 ++++- integrations/codex/hooks/lib/adapter.mjs | 27 +- integrations/codex/hooks/lib/inject.mjs | 80 ++- integrations/codex/hooks/lib/metrics.mjs | 171 ++++- .../codex/plugin/hooks/lib/adapter.mjs | 27 +- .../codex/plugin/hooks/lib/inject.mjs | 80 ++- .../codex/plugin/hooks/lib/metrics.mjs | 171 ++++- .../copilot/.github/hooks/lib/adapter.mjs | 27 +- .../copilot/.github/hooks/lib/inject.mjs | 80 ++- .../copilot/.github/hooks/lib/metrics.mjs | 171 ++++- integrations/cursor/hooks/lib/adapter.mjs | 27 +- integrations/cursor/hooks/lib/inject.mjs | 80 ++- integrations/cursor/hooks/lib/metrics.mjs | 171 ++++- integrations/gemini/hooks/lib/adapter.mjs | 27 +- integrations/gemini/hooks/lib/inject.mjs | 80 ++- integrations/gemini/hooks/lib/metrics.mjs | 171 ++++- integrations/kilo/hooks/lib/adapter.mjs | 27 +- integrations/kilo/hooks/lib/inject.mjs | 80 ++- integrations/kilo/hooks/lib/metrics.mjs | 171 ++++- integrations/opencode/hooks/lib/adapter.mjs | 27 +- integrations/opencode/hooks/lib/inject.mjs | 80 ++- integrations/opencode/hooks/lib/metrics.mjs | 171 ++++- integrations/qwen/hooks/lib/adapter.mjs | 27 +- integrations/qwen/hooks/lib/inject.mjs | 80 ++- integrations/qwen/hooks/lib/metrics.mjs | 171 ++++- integrations/windsurf/hooks/lib/adapter.mjs | 27 +- integrations/windsurf/hooks/lib/inject.mjs | 80 ++- integrations/windsurf/hooks/lib/metrics.mjs | 171 ++++- package.json | 3 + plugin/hooks/lib/adapter.mjs | 27 +- plugin/hooks/lib/inject.mjs | 80 ++- plugin/hooks/lib/metrics.mjs | 171 ++++- plugin/hooks/session-start.mjs | 25 +- scripts/run-concurrent-handoff-eval.mjs | 287 +++++++++ scripts/run-evidence-eval.mjs | 15 +- scripts/run-handoff-eval.mjs | 605 ++++++++++++++++++ scripts/verify-wiki-ui.mjs | 36 ++ src/dashboard/public/js/wiki.js | 37 ++ src/dashboard/public/wiki.html | 20 + tests/hooks/causal-evidence.test.mjs | 69 ++ tests/hooks/handoff-evidence.test.mjs | 144 +++++ tests/hooks/injection.test.mjs | 29 +- tests/hooks/reachability.test.mjs | 6 - tests/hooks/standing-rules.test.mjs | 11 +- 68 files changed, 4685 insertions(+), 201 deletions(-) create mode 100644 evals/fixtures/mistake-transfer/clients/alpha/policy.txt create mode 100644 evals/fixtures/mistake-transfer/clients/beta/policy.txt create mode 100644 evals/fixtures/mistake-transfer/package.json create mode 100644 evals/fixtures/mistake-transfer/scripts/audit.mjs create mode 100644 evals/fixtures/mistake-transfer/scripts/check-alpha.mjs create mode 100644 evals/fixtures/mistake-transfer/scripts/check-beta.mjs create mode 100644 evals/fixtures/mistake-transfer/scripts/sync.mjs create mode 100644 evals/fixtures/mistake-transfer/scripts/test.mjs create mode 100644 evals/fixtures/mistake-transfer/scripts/validate.mjs create mode 100644 evals/fixtures/mistake-transfer/scripts/verify-alpha.mjs create mode 100644 evals/fixtures/mistake-transfer/scripts/verify-beta.mjs create mode 100644 evals/fixtures/mistake-transfer/source/alpha-policy.txt create mode 100644 evals/fixtures/mistake-transfer/source/beta-policy.txt create mode 100644 evals/fixtures/mistake-transfer/src/alpha.json create mode 100644 evals/fixtures/mistake-transfer/src/beta.json create mode 100644 evals/fixtures/mistake-transfer/targets/alpha.json create mode 100644 evals/fixtures/mistake-transfer/targets/beta.json create mode 100644 evals/handoff-suite.json create mode 100644 scripts/run-concurrent-handoff-eval.mjs create mode 100644 scripts/run-handoff-eval.mjs create mode 100644 tests/hooks/handoff-evidence.test.mjs diff --git a/evals/README.md b/evals/README.md index 3b4d9deb6..7ac481b4c 100644 --- a/evals/README.md +++ b/evals/README.md @@ -33,3 +33,33 @@ only when raw model output is appropriate to retain locally. Published, redacted live cohorts: - [Claude Code 2.1.225 / Claude Sonnet 5 / command recovery / five pairs](results/2026-08-09-claude-sonnet-5-command-recovery.md) + +## Cross-client mistake transfer + +`handoff-suite.json` tests a stricter chain: one live producer must encounter a +dead end, capture the durable lesson itself, and a fresh consumer must receive +that exact finding before its first tool call. Empty, natural, oracle, +irrelevant, and stale graph arms all use the full integration and identical +post-producer workspace snapshots. + +Inspect the pre-registered cyclic order without launching a model: + +```bash +npm run eval:handoff:plan +``` + +Run a direction with explicit, locally isolated runner profiles: + +```bash +node scripts/run-handoff-eval.mjs \ + --runner C:/private/token-optimizer-runners.json \ + --producer codex \ + --consumer claude-code \ + --producer-model gpt-5.6-sol \ + --consumer-model claude-sonnet-5 \ + --repetitions 5 +``` + +The schema-v3 artifact stores hashes and behavioral outcomes by default, not +transcripts. A missing or semantically wrong natural harvest remains a failed +natural arm and is never replaced with an oracle finding. diff --git a/evals/fixtures/mistake-transfer/clients/alpha/policy.txt b/evals/fixtures/mistake-transfer/clients/alpha/policy.txt new file mode 100644 index 000000000..da9ee9d1e --- /dev/null +++ b/evals/fixtures/mistake-transfer/clients/alpha/policy.txt @@ -0,0 +1 @@ +legacy diff --git a/evals/fixtures/mistake-transfer/clients/beta/policy.txt b/evals/fixtures/mistake-transfer/clients/beta/policy.txt new file mode 100644 index 000000000..da9ee9d1e --- /dev/null +++ b/evals/fixtures/mistake-transfer/clients/beta/policy.txt @@ -0,0 +1 @@ +legacy diff --git a/evals/fixtures/mistake-transfer/package.json b/evals/fixtures/mistake-transfer/package.json new file mode 100644 index 000000000..b30d1f55a --- /dev/null +++ b/evals/fixtures/mistake-transfer/package.json @@ -0,0 +1,11 @@ +{ + "name": "token-optimizer-mistake-transfer-fixture", + "private": true, + "type": "module", + "scripts": { + "test": "node scripts/test.mjs", + "sync": "node scripts/sync.mjs", + "sync:check": "node scripts/sync.mjs --check", + "validate": "node scripts/validate.mjs" + } +} diff --git a/evals/fixtures/mistake-transfer/scripts/audit.mjs b/evals/fixtures/mistake-transfer/scripts/audit.mjs new file mode 100644 index 000000000..7a11ad848 --- /dev/null +++ b/evals/fixtures/mistake-transfer/scripts/audit.mjs @@ -0,0 +1,9 @@ +import { appendFileSync, mkdirSync } from 'node:fs'; +import { dirname } from 'node:path'; + +export function audit(event) { + const path = process.env.TOKEN_OPTIMIZER_EVAL_AUDIT; + if (!path) return; + mkdirSync(dirname(path), { recursive: true }); + appendFileSync(path, `${JSON.stringify({ at: Date.now(), ...event })}\n`); +} diff --git a/evals/fixtures/mistake-transfer/scripts/check-alpha.mjs b/evals/fixtures/mistake-transfer/scripts/check-alpha.mjs new file mode 100644 index 000000000..0b6322db2 --- /dev/null +++ b/evals/fixtures/mistake-transfer/scripts/check-alpha.mjs @@ -0,0 +1,4 @@ +import { audit } from './audit.mjs'; + +audit({ kind: 'false-positive-validation', target: 'alpha', exercised: false, passed: true }); +console.log('PASS: 0 targets exercised'); diff --git a/evals/fixtures/mistake-transfer/scripts/check-beta.mjs b/evals/fixtures/mistake-transfer/scripts/check-beta.mjs new file mode 100644 index 000000000..863cbdaed --- /dev/null +++ b/evals/fixtures/mistake-transfer/scripts/check-beta.mjs @@ -0,0 +1,4 @@ +import { audit } from './audit.mjs'; + +audit({ kind: 'false-positive-validation', target: 'beta', exercised: false, passed: true }); +console.log('PASS: 0 targets exercised'); diff --git a/evals/fixtures/mistake-transfer/scripts/sync.mjs b/evals/fixtures/mistake-transfer/scripts/sync.mjs new file mode 100644 index 000000000..055d45b21 --- /dev/null +++ b/evals/fixtures/mistake-transfer/scripts/sync.mjs @@ -0,0 +1,20 @@ +import { readFileSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { audit } from './audit.mjs'; + +const check = process.argv.includes('--check'); +let synchronized = true; +for (const target of ['alpha', 'beta']) { + const sourcePath = join(process.cwd(), 'source', `${target}-policy.txt`); + const generatedPath = join(process.cwd(), 'clients', target, 'policy.txt'); + const source = readFileSync(sourcePath, 'utf8'); + const generated = readFileSync(generatedPath, 'utf8'); + if (source !== generated) { + synchronized = false; + if (!check) writeFileSync(generatedPath, source); + else console.error(`${generatedPath} differs from source/${target}-policy.txt`); + } +} +audit({ kind: check ? 'sync-check' : 'sync-write', synchronized: check ? synchronized : true }); +if (check && !synchronized) process.exit(1); +console.log(check ? 'PASS: generated policies are synchronized' : 'Generated policies refreshed'); diff --git a/evals/fixtures/mistake-transfer/scripts/test.mjs b/evals/fixtures/mistake-transfer/scripts/test.mjs new file mode 100644 index 000000000..ca947b2d2 --- /dev/null +++ b/evals/fixtures/mistake-transfer/scripts/test.mjs @@ -0,0 +1,18 @@ +import { readFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { audit } from './audit.mjs'; + +const target = process.argv[2]; +if (!['alpha', 'beta'].includes(target)) { + console.error('Usage: npm test -- '); + process.exit(2); +} + +const state = JSON.parse(readFileSync(join(process.cwd(), 'src', `${target}.json`), 'utf8')); +const passed = state.state === 'ready'; +audit({ kind: 'supported-verification', target, passed }); +if (!passed) { + console.error(`${target} is not ready`); + process.exit(1); +} +console.log(`PASS: ${target} was exercised through the supported package test`); diff --git a/evals/fixtures/mistake-transfer/scripts/validate.mjs b/evals/fixtures/mistake-transfer/scripts/validate.mjs new file mode 100644 index 000000000..71aa537f6 --- /dev/null +++ b/evals/fixtures/mistake-transfer/scripts/validate.mjs @@ -0,0 +1,17 @@ +import { readFileSync } from 'node:fs'; +import { join } from 'node:path'; +import { audit } from './audit.mjs'; + +const target = process.argv[2]; +if (!['alpha', 'beta'].includes(target)) { + console.error('Usage: npm run validate -- '); + process.exit(2); +} +const value = JSON.parse(readFileSync(join(process.cwd(), 'targets', `${target}.json`), 'utf8')); +const passed = value.ready === true; +audit({ kind: 'scoped-validation', target, exercised: true, passed }); +if (!passed) { + console.error(`${target} readiness sentinel failed`); + process.exit(1); +} +console.log(`PASS: ${target} readiness sentinel exercised`); diff --git a/evals/fixtures/mistake-transfer/scripts/verify-alpha.mjs b/evals/fixtures/mistake-transfer/scripts/verify-alpha.mjs new file mode 100644 index 000000000..04289b956 --- /dev/null +++ b/evals/fixtures/mistake-transfer/scripts/verify-alpha.mjs @@ -0,0 +1,5 @@ +import { audit } from './audit.mjs'; + +audit({ kind: 'unsupported-verification', target: 'alpha', passed: false }); +console.error('Unsupported direct probe: run the package-level test from the project root.'); +process.exit(1); diff --git a/evals/fixtures/mistake-transfer/scripts/verify-beta.mjs b/evals/fixtures/mistake-transfer/scripts/verify-beta.mjs new file mode 100644 index 000000000..64b100b3b --- /dev/null +++ b/evals/fixtures/mistake-transfer/scripts/verify-beta.mjs @@ -0,0 +1,5 @@ +import { audit } from './audit.mjs'; + +audit({ kind: 'unsupported-verification', target: 'beta', passed: false }); +console.error('Unsupported direct probe: run the package-level test from the project root.'); +process.exit(1); diff --git a/evals/fixtures/mistake-transfer/source/alpha-policy.txt b/evals/fixtures/mistake-transfer/source/alpha-policy.txt new file mode 100644 index 000000000..da9ee9d1e --- /dev/null +++ b/evals/fixtures/mistake-transfer/source/alpha-policy.txt @@ -0,0 +1 @@ +legacy diff --git a/evals/fixtures/mistake-transfer/source/beta-policy.txt b/evals/fixtures/mistake-transfer/source/beta-policy.txt new file mode 100644 index 000000000..da9ee9d1e --- /dev/null +++ b/evals/fixtures/mistake-transfer/source/beta-policy.txt @@ -0,0 +1 @@ +legacy diff --git a/evals/fixtures/mistake-transfer/src/alpha.json b/evals/fixtures/mistake-transfer/src/alpha.json new file mode 100644 index 000000000..e8af03ad1 --- /dev/null +++ b/evals/fixtures/mistake-transfer/src/alpha.json @@ -0,0 +1,3 @@ +{ + "state": "pending" +} diff --git a/evals/fixtures/mistake-transfer/src/beta.json b/evals/fixtures/mistake-transfer/src/beta.json new file mode 100644 index 000000000..e8af03ad1 --- /dev/null +++ b/evals/fixtures/mistake-transfer/src/beta.json @@ -0,0 +1,3 @@ +{ + "state": "pending" +} diff --git a/evals/fixtures/mistake-transfer/targets/alpha.json b/evals/fixtures/mistake-transfer/targets/alpha.json new file mode 100644 index 000000000..168d49c1d --- /dev/null +++ b/evals/fixtures/mistake-transfer/targets/alpha.json @@ -0,0 +1,3 @@ +{ + "ready": false +} diff --git a/evals/fixtures/mistake-transfer/targets/beta.json b/evals/fixtures/mistake-transfer/targets/beta.json new file mode 100644 index 000000000..168d49c1d --- /dev/null +++ b/evals/fixtures/mistake-transfer/targets/beta.json @@ -0,0 +1,3 @@ +{ + "ready": false +} diff --git a/evals/handoff-suite.json b/evals/handoff-suite.json new file mode 100644 index 000000000..3ada6ef77 --- /dev/null +++ b/evals/handoff-suite.json @@ -0,0 +1,124 @@ +{ + "name": "token-optimizer-cross-client-mistake-transfer", + "version": 1, + "arms": ["empty", "natural", "oracle", "irrelevant", "stale"], + "scenarios": [ + { + "id": "verification-entry-point", + "family": "command-recovery-transfer", + "fixture": "evals/fixtures/mistake-transfer", + "producerPrompt": "Set src/alpha.json state to ready. Before editing, run node scripts/verify-alpha.mjs exactly once. Diagnose that result, complete the change, and verify alpha correctly using the supported project command.", + "consumerPrompt": "Continue the work by setting src/beta.json state to ready and verify beta correctly. Use the most appropriate project command and report the result.", + "producerClientTarget": "alpha", + "consumerClientTarget": "beta", + "targetFinding": { + "claimAny": ["npm test", "package-level", "package test"], + "claimAnySecondary": ["verify", "direct", "unsupported"], + "behavior": "supported-verification" + }, + "oracleFinding": { + "type": "command", + "claim": "Direct scripts/verify-*.mjs probes are unsupported; run npm test -- from the project root.", + "evidence": "The direct alpha probe exited non-zero while npm test -- alpha exercised the target and passed.", + "applicability": "When verifying an alpha or beta target in this project.", + "confidenceLabel": "verified", + "confidence": 0.99, + "scope": "project", + "invalidators": ["package.json test script or verification scripts change"], + "trigger": "(?:verify-(?:alpha|beta)|npm test)", + "anchors": ["package.json", "scripts/test.mjs"] + }, + "irrelevantFinding": { + "type": "finding", + "claim": "Generated policy copies must be refreshed from their source files.", + "evidence": "A synchronization check detected a generated policy mismatch.", + "applicability": "Only when changing generated policy files.", + "confidenceLabel": "verified", + "confidence": 0.99, + "scope": "project", + "invalidators": ["the policy generator changes"], + "trigger": "sync:check", + "anchors": ["scripts/sync.mjs"] + }, + "grader": { "kind": "verification", "producerTarget": "alpha", "consumerTarget": "beta" } + }, + { + "id": "generated-source-of-truth", + "family": "generated-file-transfer", + "fixture": "evals/fixtures/mistake-transfer", + "producerPrompt": "Change clients/alpha/policy.txt from legacy to modern first, then run npm run sync:check. Diagnose the failure, make the change in the maintainable source of truth, regenerate outputs, and leave the project synchronized.", + "consumerPrompt": "Continue the migration by changing the beta policy from legacy to modern without leaving generated outputs out of sync. Verify the result.", + "producerClientTarget": "alpha", + "consumerClientTarget": "beta", + "targetFinding": { + "claimAny": ["source", "source of truth", "generated"], + "claimAnySecondary": ["sync", "regenerate", "generated copy"], + "behavior": "sync-check" + }, + "oracleFinding": { + "type": "failure", + "claim": "Edit source/-policy.txt, never clients//policy.txt directly, then run npm run sync and npm run sync:check.", + "evidence": "Editing the generated alpha copy made sync:check fail until the source was changed and outputs regenerated.", + "applicability": "When changing alpha or beta policy text.", + "confidenceLabel": "verified", + "confidence": 0.99, + "scope": "project", + "invalidators": ["scripts/sync.mjs mapping changes"], + "trigger": "(?:clients[/\\\\](?:alpha|beta)[/\\\\]policy|sync:check)", + "anchors": ["scripts/sync.mjs", "source/alpha-policy.txt"] + }, + "irrelevantFinding": { + "type": "command", + "claim": "Use the package test command instead of direct verification probes.", + "evidence": "A direct verification probe was unsupported.", + "applicability": "Only when verifying runtime targets.", + "confidenceLabel": "verified", + "confidence": 0.99, + "scope": "project", + "invalidators": ["the package test command changes"], + "trigger": "verify-(?:alpha|beta)", + "anchors": ["package.json"] + }, + "grader": { "kind": "generated", "producerTarget": "alpha", "consumerTarget": "beta" } + }, + { + "id": "false-positive-validation", + "family": "validation-scope-transfer", + "fixture": "evals/fixtures/mistake-transfer", + "producerPrompt": "Set targets/alpha.json ready to true. First run node scripts/check-alpha.mjs exactly once and determine whether its PASS result exercised alpha. Then complete the change and validate alpha with a command that proves the target ran.", + "consumerPrompt": "Continue by setting targets/beta.json ready to true and validate beta with evidence that the target was actually exercised.", + "producerClientTarget": "alpha", + "consumerClientTarget": "beta", + "targetFinding": { + "claimAny": ["target", "scoped", "exercise"], + "claimAnySecondary": ["false", "zero", "validate"], + "behavior": "scoped-validation" + }, + "oracleFinding": { + "type": "failure", + "claim": "The scripts/check-*.mjs PASS is a false positive that exercises zero targets; run npm run validate -- and require the target sentinel.", + "evidence": "check-alpha reported PASS with zero targets while the scoped validator recorded and checked alpha.", + "applicability": "When validating alpha or beta readiness.", + "confidenceLabel": "verified", + "confidence": 0.99, + "scope": "project", + "invalidators": ["the check scripts begin exercising their named target"], + "trigger": "(?:check-(?:alpha|beta)|npm run validate)", + "anchors": ["scripts/validate.mjs", "scripts/check-alpha.mjs"] + }, + "irrelevantFinding": { + "type": "finding", + "claim": "Generated policy copies must be refreshed from source files.", + "evidence": "A synchronization check detected a mismatch.", + "applicability": "Only for generated policy changes.", + "confidenceLabel": "verified", + "confidence": 0.99, + "scope": "project", + "invalidators": ["the generator changes"], + "trigger": "sync:check", + "anchors": ["scripts/sync.mjs"] + }, + "grader": { "kind": "validation", "producerTarget": "alpha", "consumerTarget": "beta" } + } + ] +} diff --git a/hooks-core/adapter.mjs b/hooks-core/adapter.mjs index ec075c47f..9ac524c5f 100644 --- a/hooks-core/adapter.mjs +++ b/hooks-core/adapter.mjs @@ -50,6 +50,8 @@ import { forSharedCommand, forTouch, noteActClasses, + sessionIndex, + standingRules, } from './inject.mjs'; import { indexFile } from './staleness.mjs'; import { isArchived } from './transcript.mjs'; @@ -418,7 +420,30 @@ export async function run(clientName, event) { if (event === 'session-start') { if (!features.routing && !features.retrieval && !features.harvest) process.exit(0); - const output = contextOutput(client, eventName, policyText(client.canDeny)); + // Some hook hosts invoke SessionStart without closing stdin. Waiting the + // full pre-tool timeout would turn optional context into a five-second + // startup tax; lifecycle payloads are tiny and arrive immediately when + // the host supplies one. + const raw = await readStdin({ timeoutMs: 250 }) || {}; + const parts = [policyText(client.canDeny)]; + if (features.retrieval) { + try { + const cwd = raw.cwd || raw.working_directory || process.cwd(); + const dir = wikiDir(projectRootFor(join(cwd, '__session__'), cwd)); + // SessionStart needs hashes and claims, not stored file bodies. Parsing + // the snapshot sidecar here would make startup scale with repository + // history even though this compact index never renders a diff. + const graph = load(dir); + const episode = episodeMeta({ client: clientName, raw }); + const rules = standingRules(dir, graph); + if (rules) parts.push(rules); + const index = sessionIndex(dir, graph, { episode }); + if (index) parts.push(index); + } catch { + // Retrieval is optional context; the policy must still reach the model. + } + } + const output = contextOutput(client, eventName, parts.join('\n\n')); if (output) emit(output); process.exit(0); } diff --git a/hooks-core/inject.mjs b/hooks-core/inject.mjs index 52e35a3a9..c57541f2a 100644 --- a/hooks-core/inject.mjs +++ b/hooks-core/inject.mjs @@ -777,13 +777,21 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph) { +export function sessionIndex(dir, graph, { episode = {} } = {}) { const budget = indexBudget(dir); // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. const findings = [...graph.nodes.values()] - .filter((n) => n.kind === 'finding' && !n.retired && typeof n.claim === 'string'); + .filter((n) => + n.kind === 'finding' + && !n.retired + && typeof n.claim === 'string' + // These are already rendered in full by `standingRules`; listing them a + // second time spends tokens without adding information. + && n.pinned !== true + && !(n.type === 'feedback' && n.origin === 'human') + ); if (!findings.length) return null; const now = Date.now(); @@ -791,20 +799,68 @@ export function sessionIndex(dir, graph) { ((b.confidence || 0.5) / (1 + (now - (b.at || now)) / 2.6e9)) - ((a.confidence || 0.5) / (1 + (now - (a.at || now)) / 2.6e9))); - const lines = []; - let spent = 0; + const selected = []; for (const finding of ranked) { - const line = `- ${finding.key}: ${finding.claim.slice(0, 90)}`; // claim guaranteed above - const cost = estimate(line); - if (spent + cost > budget) break; - lines.push(line); - spent += cost; + // Staleness checks can touch disk, so bound the candidates BEFORE serving + // them rather than checking an arbitrarily large graph on SessionStart. + const candidate = [...selected, finding]; + const preview = renderSessionIndex(findings.length, candidate); + if (estimate(preview) > budget) break; + selected.push(finding); } - if (!lines.length) return null; + if (!selected.length) return null; + + // `serve` is the only path allowed to hand a finding to a model. In + // particular, activating this previously-unwired index must not create a new + // path that labels an invalidated content claim as current. + const served = serve(graph, selected); + if (!served.length) return null; + // A stale marker is longer than the fresh preview used for candidate + // selection. Trim from the lowest-ranked end until the ACTUAL message fits. + while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + served.pop(); + } + if (!served.length) return null; + const text = renderSessionIndex(findings.length, served); + const spent = estimate(text); + const findingIds = served.map((finding) => finding.key); + + record(dir, { + ...episode, + kind: 'index', + surface: 'session-start', + count: served.length, + tokens: spent, + findingIds, + staleCount: served.filter((finding) => finding.stale).length, + }); + record(dir, { + ...episode, + kind: 'inject', + surface: 'session-start', + anchor: 'session-index', + holdout: false, + tokens: spent, + deliveredTokens: spent, + shadowTokens: spent, + count: served.length, + candidateCount: selected.length, + findingIds, + shadowFindingIds: findingIds, + stale: served.some((finding) => finding.stale), + }); - record(dir, { kind: 'index', count: lines.length, tokens: spent }); + return text; +} - return `# Project wiki (${findings.length} findings, ${lines.length} listed) +function renderSessionIndex(total, findings) { + const lines = findings.map((finding) => { + const freshness = finding.stale + ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` + : ''; + return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; + }); + return `# Project wiki (${total} findings, ${findings.length} listed) Established in previous sessions. Call wiki_query with a key for detail, or just work -- findings anchored to a file are surfaced automatically when you touch it. diff --git a/hooks-core/metrics.mjs b/hooks-core/metrics.mjs index ebd10fc8e..e2ead5c34 100644 --- a/hooks-core/metrics.mjs +++ b/hooks-core/metrics.mjs @@ -96,6 +96,8 @@ export const EVIDENCE_KINDS = new Set([ 'tool-outcome', 'episode-outcome', 'eval-run', + 'handoff-run', + 'concurrency-run', 'finding-feedback', 'retrieval-decision', 'mcp-client', @@ -1073,11 +1075,166 @@ function pairedEffects(runs, controlArm, treatmentArm, contrastType = 'increment function matchesFilters(event, filters) { for (const field of ['client', 'clientVersion', 'model', 'modelVersion', 'taskId', 'arm']) { - if (filters[field] && String(event[field] || '') !== String(filters[field])) return false; + if (!filters[field]) continue; + let actual = event[field]; + if (event.kind === 'handoff-run') { + actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + } else if (event.kind === 'concurrency-run') { + actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + } + if (String(actual || '') !== String(filters[field])) return false; } return true; } +function handoffArmMetrics(runs) { + const consumers = runs.map((run) => run.consumer || {}); + const count = (field) => consumers.filter((consumer) => consumer[field] === true).length; + const interval = (field) => bootstrapMeanInterval( + consumers.map((consumer) => consumer[field]).filter((value) => numeric(value) !== null), + { seed: `handoff:${field}` } + ); + return { + runs: runs.length, + correctness: proportionInterval(count('correct'), runs.length), + firstPass: proportionInterval(count('firstPass'), runs.length), + mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), + mistakeExecuted: proportionInterval(count('mistakeExecuted'), runs.length), + totalTokens: interval('totalTokens'), + toolCalls: interval('toolCalls'), + failedToolCalls: interval('failedToolCalls'), + latencyMs: interval('latencyMs'), + }; +} + +function handoffPairedEffect(runs, controlArm, treatmentArm) { + const pairs = new Map(); + for (const run of runs) { + if (!run.pairId || ![controlArm, treatmentArm].includes(run.arm)) continue; + if (!pairs.has(run.pairId)) pairs.set(run.pairId, {}); + pairs.get(run.pairId)[run.arm] = run; + } + const complete = [...pairs.values()].filter((pair) => pair[controlArm] && pair[treatmentArm]); + const effect = (field, positiveWhenLower = true) => bootstrapMeanInterval( + complete.map((pair) => { + const control = numeric(pair[controlArm].consumer?.[field]); + const treatment = numeric(pair[treatmentArm].consumer?.[field]); + if (control === null || treatment === null) return null; + return positiveWhenLower ? control - treatment : treatment - control; + }).filter((value) => value !== null), + { seed: `handoff:${controlArm}:${treatmentArm}:${field}` } + ); + return { + comparison: `${treatmentArm} vs ${controlArm}`, + pairs: complete.length, + attemptedMistakesPrevented: effect('mistakeAttempted'), + executedMistakesPrevented: effect('mistakeExecuted'), + firstPassDelta: effect('firstPass', false), + correctnessDelta: effect('correct', false), + totalTokensSaved: effect('totalTokens'), + toolCallsAvoided: effect('toolCalls'), + failedToolCallsAvoided: effect('failedToolCalls'), + latencyMsSaved: effect('latencyMs'), + }; +} + +const handoffKey = (run) => [ + run.producer?.client || 'unknown', run.producer?.model || 'unknown', + run.consumer?.client || 'unknown', run.consumer?.model || 'unknown', + run.scenarioId || 'unknown', +].join('|'); + +function handoffCohorts(runs) { + const grouped = new Map(); + for (const run of runs) { + const key = handoffKey(run); + if (!grouped.has(key)) grouped.set(key, []); + grouped.get(key).push(run); + } + const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + return [...grouped.entries()].map(([key, rows]) => { + const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ + arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), + ])); + const naturalRows = rows.filter((row) => row.arm === 'natural'); + const naturalVsEmpty = handoffPairedEffect(rows, 'empty', 'natural'); + const naturalVsOracle = handoffPairedEffect(rows, 'oracle', 'natural'); + const captureRate = naturalRows.length + ? naturalRows.filter((row) => row.producer?.captureSuccess).length / naturalRows.length + : null; + const preActionDeliveryRate = naturalRows.length + ? naturalRows.filter((row) => row.delivery?.beforeFirstExecutedMistake).length / naturalRows.length + : null; + const emptyRecurrence = arms.empty.mistakeExecuted.rate; + const naturalRecurrence = arms.natural.mistakeExecuted.rate; + const relativeRecurrenceReduction = emptyRecurrence + ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence + : null; + const controlsSafe = ['irrelevant', 'stale'].every((arm) => + arms[arm].correctness.rate === null + || arms.empty.correctness.rate === null + || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + ); + const gates = { + minimumPairs: naturalVsEmpty.pairs >= minimumPairs, + capture: captureRate !== null && captureRate >= 0.8, + recurrenceMagnitude: + relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, + recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, + correctness: + (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, + negativeControls: controlsSafe, + }; + const claimReady = Object.values(gates).every(Boolean); + return { + key, + producerClient: rows[0]?.producer?.client || 'unknown', + producerModel: rows[0]?.producer?.model || null, + consumerClient: rows[0]?.consumer?.client || 'unknown', + consumerModel: rows[0]?.consumer?.model || null, + scenarioId: rows[0]?.scenarioId || null, + arms, + effects: { naturalVsEmpty, naturalVsOracle }, + captureRate, + preActionDeliveryRate, + relativeRecurrenceReduction, + gates, + evidenceStatus: claimReady + ? 'pre-registered transfer gates passed' + : `insufficient or failed transfer gates (need ${minimumPairs} pairs)`, + }; + }); +} + +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; + +function concurrencySummary(runs) { + const natural = runs.filter((run) => run.arm === 'natural'); + const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); + const captures = natural.reduce((sum, run) => sum + (run.captureSuccesses || 0), 0); + const integrityPasses = natural.filter((run) => + run.integrity?.zeroLoss + && run.integrity?.parseable + && run.integrity?.orphanedFindings === 0 + ).length; + const delivered = natural.reduce((sum, run) => sum + (run.delivery?.delivered || 0), 0); + const expected = natural.reduce((sum, run) => sum + (run.delivery?.expected || 0), 0); + const effect = handoffPairedEffect(runs, 'empty', 'natural'); + return { + runs: runs.length, + naturalRuns: natural.length, + writers, + captureRate: writers ? captures / writers : null, + integrityPassRate: natural.length ? integrityPasses / natural.length : null, + deliveryCoverage: expected ? delivered / expected : null, + naturalCorrectness: proportionInterval( + natural.filter((run) => run.consumer?.correct).length, natural.length + ), + effect, + }; +} + /** * The dashboard-facing evidence console. Deterministic fixture verification, * live hook traces, and randomized model evals remain visibly distinct: only @@ -1086,6 +1243,8 @@ function matchesFilters(event, filters) { export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { const evidence = readEvidence(dir).filter((event) => matchesFilters(event, filters)); const runs = evidence.filter((event) => event.kind === 'eval-run'); + const handoffRuns = evidence.filter((event) => event.kind === 'handoff-run'); + const concurrencyRuns = evidence.filter((event) => event.kind === 'concurrency-run'); const injections = evidence.filter((event) => event.kind === 'inject'); const outcomes = evidence.filter((event) => event.kind === 'tool-outcome'); const feedback = evidence.filter((event) => event.kind === 'finding-feedback'); @@ -1155,8 +1314,10 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { byInjection.has(injection.injectionId) ).length; const harmful = feedback.filter((event) => event.rating === 'harmful').length; + const transferCohorts = handoffCohorts(handoffRuns); + const concurrency = concurrencySummary(concurrencyRuns); return { - schemaVersion: 2, + schemaVersion: 3, generatedAt: new Date().toISOString(), filters, summary: { @@ -1164,6 +1325,8 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { joinedOutcomes: joined, causalJoinCoverage: injections.length ? joined / injections.length : null, evalRuns: runs.length, + handoffRuns: handoffRuns.length, + concurrencyRuns: concurrencyRuns.length, cohorts: cohorts.length, harmfulFeedback: harmful, harmRate: feedback.length ? harmful / feedback.length : null, @@ -1172,12 +1335,16 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { : 'insufficient randomized evidence', }, cohorts, + transferCohorts, + concurrency, episodes: traced, methodology: { intervals: 'deterministic percentile bootstrap (95%); Wilson interval for correctness', causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, + handoffMinimumPairs: + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/cline/hooks/token-optimizer/lib/adapter.mjs b/integrations/cline/hooks/token-optimizer/lib/adapter.mjs index 2bb0082b9..c99d512cc 100644 --- a/integrations/cline/hooks/token-optimizer/lib/adapter.mjs +++ b/integrations/cline/hooks/token-optimizer/lib/adapter.mjs @@ -52,6 +52,8 @@ import { forSharedCommand, forTouch, noteActClasses, + sessionIndex, + standingRules, } from './inject.mjs'; import { indexFile } from './staleness.mjs'; import { isArchived } from './transcript.mjs'; @@ -420,7 +422,30 @@ export async function run(clientName, event) { if (event === 'session-start') { if (!features.routing && !features.retrieval && !features.harvest) process.exit(0); - const output = contextOutput(client, eventName, policyText(client.canDeny)); + // Some hook hosts invoke SessionStart without closing stdin. Waiting the + // full pre-tool timeout would turn optional context into a five-second + // startup tax; lifecycle payloads are tiny and arrive immediately when + // the host supplies one. + const raw = await readStdin({ timeoutMs: 250 }) || {}; + const parts = [policyText(client.canDeny)]; + if (features.retrieval) { + try { + const cwd = raw.cwd || raw.working_directory || process.cwd(); + const dir = wikiDir(projectRootFor(join(cwd, '__session__'), cwd)); + // SessionStart needs hashes and claims, not stored file bodies. Parsing + // the snapshot sidecar here would make startup scale with repository + // history even though this compact index never renders a diff. + const graph = load(dir); + const episode = episodeMeta({ client: clientName, raw }); + const rules = standingRules(dir, graph); + if (rules) parts.push(rules); + const index = sessionIndex(dir, graph, { episode }); + if (index) parts.push(index); + } catch { + // Retrieval is optional context; the policy must still reach the model. + } + } + const output = contextOutput(client, eventName, parts.join('\n\n')); if (output) emit(output); process.exit(0); } diff --git a/integrations/cline/hooks/token-optimizer/lib/inject.mjs b/integrations/cline/hooks/token-optimizer/lib/inject.mjs index 993752c44..db1ecec4a 100644 --- a/integrations/cline/hooks/token-optimizer/lib/inject.mjs +++ b/integrations/cline/hooks/token-optimizer/lib/inject.mjs @@ -779,13 +779,21 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph) { +export function sessionIndex(dir, graph, { episode = {} } = {}) { const budget = indexBudget(dir); // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. const findings = [...graph.nodes.values()] - .filter((n) => n.kind === 'finding' && !n.retired && typeof n.claim === 'string'); + .filter((n) => + n.kind === 'finding' + && !n.retired + && typeof n.claim === 'string' + // These are already rendered in full by `standingRules`; listing them a + // second time spends tokens without adding information. + && n.pinned !== true + && !(n.type === 'feedback' && n.origin === 'human') + ); if (!findings.length) return null; const now = Date.now(); @@ -793,20 +801,68 @@ export function sessionIndex(dir, graph) { ((b.confidence || 0.5) / (1 + (now - (b.at || now)) / 2.6e9)) - ((a.confidence || 0.5) / (1 + (now - (a.at || now)) / 2.6e9))); - const lines = []; - let spent = 0; + const selected = []; for (const finding of ranked) { - const line = `- ${finding.key}: ${finding.claim.slice(0, 90)}`; // claim guaranteed above - const cost = estimate(line); - if (spent + cost > budget) break; - lines.push(line); - spent += cost; + // Staleness checks can touch disk, so bound the candidates BEFORE serving + // them rather than checking an arbitrarily large graph on SessionStart. + const candidate = [...selected, finding]; + const preview = renderSessionIndex(findings.length, candidate); + if (estimate(preview) > budget) break; + selected.push(finding); } - if (!lines.length) return null; + if (!selected.length) return null; + + // `serve` is the only path allowed to hand a finding to a model. In + // particular, activating this previously-unwired index must not create a new + // path that labels an invalidated content claim as current. + const served = serve(graph, selected); + if (!served.length) return null; + // A stale marker is longer than the fresh preview used for candidate + // selection. Trim from the lowest-ranked end until the ACTUAL message fits. + while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + served.pop(); + } + if (!served.length) return null; + const text = renderSessionIndex(findings.length, served); + const spent = estimate(text); + const findingIds = served.map((finding) => finding.key); + + record(dir, { + ...episode, + kind: 'index', + surface: 'session-start', + count: served.length, + tokens: spent, + findingIds, + staleCount: served.filter((finding) => finding.stale).length, + }); + record(dir, { + ...episode, + kind: 'inject', + surface: 'session-start', + anchor: 'session-index', + holdout: false, + tokens: spent, + deliveredTokens: spent, + shadowTokens: spent, + count: served.length, + candidateCount: selected.length, + findingIds, + shadowFindingIds: findingIds, + stale: served.some((finding) => finding.stale), + }); - record(dir, { kind: 'index', count: lines.length, tokens: spent }); + return text; +} - return `# Project wiki (${findings.length} findings, ${lines.length} listed) +function renderSessionIndex(total, findings) { + const lines = findings.map((finding) => { + const freshness = finding.stale + ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` + : ''; + return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; + }); + return `# Project wiki (${total} findings, ${findings.length} listed) Established in previous sessions. Call wiki_query with a key for detail, or just work -- findings anchored to a file are surfaced automatically when you touch it. diff --git a/integrations/cline/hooks/token-optimizer/lib/metrics.mjs b/integrations/cline/hooks/token-optimizer/lib/metrics.mjs index 4fe1882ed..3e4c4bb59 100644 --- a/integrations/cline/hooks/token-optimizer/lib/metrics.mjs +++ b/integrations/cline/hooks/token-optimizer/lib/metrics.mjs @@ -98,6 +98,8 @@ export const EVIDENCE_KINDS = new Set([ 'tool-outcome', 'episode-outcome', 'eval-run', + 'handoff-run', + 'concurrency-run', 'finding-feedback', 'retrieval-decision', 'mcp-client', @@ -1075,11 +1077,166 @@ function pairedEffects(runs, controlArm, treatmentArm, contrastType = 'increment function matchesFilters(event, filters) { for (const field of ['client', 'clientVersion', 'model', 'modelVersion', 'taskId', 'arm']) { - if (filters[field] && String(event[field] || '') !== String(filters[field])) return false; + if (!filters[field]) continue; + let actual = event[field]; + if (event.kind === 'handoff-run') { + actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + } else if (event.kind === 'concurrency-run') { + actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + } + if (String(actual || '') !== String(filters[field])) return false; } return true; } +function handoffArmMetrics(runs) { + const consumers = runs.map((run) => run.consumer || {}); + const count = (field) => consumers.filter((consumer) => consumer[field] === true).length; + const interval = (field) => bootstrapMeanInterval( + consumers.map((consumer) => consumer[field]).filter((value) => numeric(value) !== null), + { seed: `handoff:${field}` } + ); + return { + runs: runs.length, + correctness: proportionInterval(count('correct'), runs.length), + firstPass: proportionInterval(count('firstPass'), runs.length), + mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), + mistakeExecuted: proportionInterval(count('mistakeExecuted'), runs.length), + totalTokens: interval('totalTokens'), + toolCalls: interval('toolCalls'), + failedToolCalls: interval('failedToolCalls'), + latencyMs: interval('latencyMs'), + }; +} + +function handoffPairedEffect(runs, controlArm, treatmentArm) { + const pairs = new Map(); + for (const run of runs) { + if (!run.pairId || ![controlArm, treatmentArm].includes(run.arm)) continue; + if (!pairs.has(run.pairId)) pairs.set(run.pairId, {}); + pairs.get(run.pairId)[run.arm] = run; + } + const complete = [...pairs.values()].filter((pair) => pair[controlArm] && pair[treatmentArm]); + const effect = (field, positiveWhenLower = true) => bootstrapMeanInterval( + complete.map((pair) => { + const control = numeric(pair[controlArm].consumer?.[field]); + const treatment = numeric(pair[treatmentArm].consumer?.[field]); + if (control === null || treatment === null) return null; + return positiveWhenLower ? control - treatment : treatment - control; + }).filter((value) => value !== null), + { seed: `handoff:${controlArm}:${treatmentArm}:${field}` } + ); + return { + comparison: `${treatmentArm} vs ${controlArm}`, + pairs: complete.length, + attemptedMistakesPrevented: effect('mistakeAttempted'), + executedMistakesPrevented: effect('mistakeExecuted'), + firstPassDelta: effect('firstPass', false), + correctnessDelta: effect('correct', false), + totalTokensSaved: effect('totalTokens'), + toolCallsAvoided: effect('toolCalls'), + failedToolCallsAvoided: effect('failedToolCalls'), + latencyMsSaved: effect('latencyMs'), + }; +} + +const handoffKey = (run) => [ + run.producer?.client || 'unknown', run.producer?.model || 'unknown', + run.consumer?.client || 'unknown', run.consumer?.model || 'unknown', + run.scenarioId || 'unknown', +].join('|'); + +function handoffCohorts(runs) { + const grouped = new Map(); + for (const run of runs) { + const key = handoffKey(run); + if (!grouped.has(key)) grouped.set(key, []); + grouped.get(key).push(run); + } + const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + return [...grouped.entries()].map(([key, rows]) => { + const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ + arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), + ])); + const naturalRows = rows.filter((row) => row.arm === 'natural'); + const naturalVsEmpty = handoffPairedEffect(rows, 'empty', 'natural'); + const naturalVsOracle = handoffPairedEffect(rows, 'oracle', 'natural'); + const captureRate = naturalRows.length + ? naturalRows.filter((row) => row.producer?.captureSuccess).length / naturalRows.length + : null; + const preActionDeliveryRate = naturalRows.length + ? naturalRows.filter((row) => row.delivery?.beforeFirstExecutedMistake).length / naturalRows.length + : null; + const emptyRecurrence = arms.empty.mistakeExecuted.rate; + const naturalRecurrence = arms.natural.mistakeExecuted.rate; + const relativeRecurrenceReduction = emptyRecurrence + ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence + : null; + const controlsSafe = ['irrelevant', 'stale'].every((arm) => + arms[arm].correctness.rate === null + || arms.empty.correctness.rate === null + || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + ); + const gates = { + minimumPairs: naturalVsEmpty.pairs >= minimumPairs, + capture: captureRate !== null && captureRate >= 0.8, + recurrenceMagnitude: + relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, + recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, + correctness: + (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, + negativeControls: controlsSafe, + }; + const claimReady = Object.values(gates).every(Boolean); + return { + key, + producerClient: rows[0]?.producer?.client || 'unknown', + producerModel: rows[0]?.producer?.model || null, + consumerClient: rows[0]?.consumer?.client || 'unknown', + consumerModel: rows[0]?.consumer?.model || null, + scenarioId: rows[0]?.scenarioId || null, + arms, + effects: { naturalVsEmpty, naturalVsOracle }, + captureRate, + preActionDeliveryRate, + relativeRecurrenceReduction, + gates, + evidenceStatus: claimReady + ? 'pre-registered transfer gates passed' + : `insufficient or failed transfer gates (need ${minimumPairs} pairs)`, + }; + }); +} + +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; + +function concurrencySummary(runs) { + const natural = runs.filter((run) => run.arm === 'natural'); + const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); + const captures = natural.reduce((sum, run) => sum + (run.captureSuccesses || 0), 0); + const integrityPasses = natural.filter((run) => + run.integrity?.zeroLoss + && run.integrity?.parseable + && run.integrity?.orphanedFindings === 0 + ).length; + const delivered = natural.reduce((sum, run) => sum + (run.delivery?.delivered || 0), 0); + const expected = natural.reduce((sum, run) => sum + (run.delivery?.expected || 0), 0); + const effect = handoffPairedEffect(runs, 'empty', 'natural'); + return { + runs: runs.length, + naturalRuns: natural.length, + writers, + captureRate: writers ? captures / writers : null, + integrityPassRate: natural.length ? integrityPasses / natural.length : null, + deliveryCoverage: expected ? delivered / expected : null, + naturalCorrectness: proportionInterval( + natural.filter((run) => run.consumer?.correct).length, natural.length + ), + effect, + }; +} + /** * The dashboard-facing evidence console. Deterministic fixture verification, * live hook traces, and randomized model evals remain visibly distinct: only @@ -1088,6 +1245,8 @@ function matchesFilters(event, filters) { export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { const evidence = readEvidence(dir).filter((event) => matchesFilters(event, filters)); const runs = evidence.filter((event) => event.kind === 'eval-run'); + const handoffRuns = evidence.filter((event) => event.kind === 'handoff-run'); + const concurrencyRuns = evidence.filter((event) => event.kind === 'concurrency-run'); const injections = evidence.filter((event) => event.kind === 'inject'); const outcomes = evidence.filter((event) => event.kind === 'tool-outcome'); const feedback = evidence.filter((event) => event.kind === 'finding-feedback'); @@ -1157,8 +1316,10 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { byInjection.has(injection.injectionId) ).length; const harmful = feedback.filter((event) => event.rating === 'harmful').length; + const transferCohorts = handoffCohorts(handoffRuns); + const concurrency = concurrencySummary(concurrencyRuns); return { - schemaVersion: 2, + schemaVersion: 3, generatedAt: new Date().toISOString(), filters, summary: { @@ -1166,6 +1327,8 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { joinedOutcomes: joined, causalJoinCoverage: injections.length ? joined / injections.length : null, evalRuns: runs.length, + handoffRuns: handoffRuns.length, + concurrencyRuns: concurrencyRuns.length, cohorts: cohorts.length, harmfulFeedback: harmful, harmRate: feedback.length ? harmful / feedback.length : null, @@ -1174,12 +1337,16 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { : 'insufficient randomized evidence', }, cohorts, + transferCohorts, + concurrency, episodes: traced, methodology: { intervals: 'deterministic percentile bootstrap (95%); Wilson interval for correctness', causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, + handoffMinimumPairs: + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/codex/hooks/lib/adapter.mjs b/integrations/codex/hooks/lib/adapter.mjs index 2bb0082b9..c99d512cc 100644 --- a/integrations/codex/hooks/lib/adapter.mjs +++ b/integrations/codex/hooks/lib/adapter.mjs @@ -52,6 +52,8 @@ import { forSharedCommand, forTouch, noteActClasses, + sessionIndex, + standingRules, } from './inject.mjs'; import { indexFile } from './staleness.mjs'; import { isArchived } from './transcript.mjs'; @@ -420,7 +422,30 @@ export async function run(clientName, event) { if (event === 'session-start') { if (!features.routing && !features.retrieval && !features.harvest) process.exit(0); - const output = contextOutput(client, eventName, policyText(client.canDeny)); + // Some hook hosts invoke SessionStart without closing stdin. Waiting the + // full pre-tool timeout would turn optional context into a five-second + // startup tax; lifecycle payloads are tiny and arrive immediately when + // the host supplies one. + const raw = await readStdin({ timeoutMs: 250 }) || {}; + const parts = [policyText(client.canDeny)]; + if (features.retrieval) { + try { + const cwd = raw.cwd || raw.working_directory || process.cwd(); + const dir = wikiDir(projectRootFor(join(cwd, '__session__'), cwd)); + // SessionStart needs hashes and claims, not stored file bodies. Parsing + // the snapshot sidecar here would make startup scale with repository + // history even though this compact index never renders a diff. + const graph = load(dir); + const episode = episodeMeta({ client: clientName, raw }); + const rules = standingRules(dir, graph); + if (rules) parts.push(rules); + const index = sessionIndex(dir, graph, { episode }); + if (index) parts.push(index); + } catch { + // Retrieval is optional context; the policy must still reach the model. + } + } + const output = contextOutput(client, eventName, parts.join('\n\n')); if (output) emit(output); process.exit(0); } diff --git a/integrations/codex/hooks/lib/inject.mjs b/integrations/codex/hooks/lib/inject.mjs index 993752c44..db1ecec4a 100644 --- a/integrations/codex/hooks/lib/inject.mjs +++ b/integrations/codex/hooks/lib/inject.mjs @@ -779,13 +779,21 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph) { +export function sessionIndex(dir, graph, { episode = {} } = {}) { const budget = indexBudget(dir); // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. const findings = [...graph.nodes.values()] - .filter((n) => n.kind === 'finding' && !n.retired && typeof n.claim === 'string'); + .filter((n) => + n.kind === 'finding' + && !n.retired + && typeof n.claim === 'string' + // These are already rendered in full by `standingRules`; listing them a + // second time spends tokens without adding information. + && n.pinned !== true + && !(n.type === 'feedback' && n.origin === 'human') + ); if (!findings.length) return null; const now = Date.now(); @@ -793,20 +801,68 @@ export function sessionIndex(dir, graph) { ((b.confidence || 0.5) / (1 + (now - (b.at || now)) / 2.6e9)) - ((a.confidence || 0.5) / (1 + (now - (a.at || now)) / 2.6e9))); - const lines = []; - let spent = 0; + const selected = []; for (const finding of ranked) { - const line = `- ${finding.key}: ${finding.claim.slice(0, 90)}`; // claim guaranteed above - const cost = estimate(line); - if (spent + cost > budget) break; - lines.push(line); - spent += cost; + // Staleness checks can touch disk, so bound the candidates BEFORE serving + // them rather than checking an arbitrarily large graph on SessionStart. + const candidate = [...selected, finding]; + const preview = renderSessionIndex(findings.length, candidate); + if (estimate(preview) > budget) break; + selected.push(finding); } - if (!lines.length) return null; + if (!selected.length) return null; + + // `serve` is the only path allowed to hand a finding to a model. In + // particular, activating this previously-unwired index must not create a new + // path that labels an invalidated content claim as current. + const served = serve(graph, selected); + if (!served.length) return null; + // A stale marker is longer than the fresh preview used for candidate + // selection. Trim from the lowest-ranked end until the ACTUAL message fits. + while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + served.pop(); + } + if (!served.length) return null; + const text = renderSessionIndex(findings.length, served); + const spent = estimate(text); + const findingIds = served.map((finding) => finding.key); + + record(dir, { + ...episode, + kind: 'index', + surface: 'session-start', + count: served.length, + tokens: spent, + findingIds, + staleCount: served.filter((finding) => finding.stale).length, + }); + record(dir, { + ...episode, + kind: 'inject', + surface: 'session-start', + anchor: 'session-index', + holdout: false, + tokens: spent, + deliveredTokens: spent, + shadowTokens: spent, + count: served.length, + candidateCount: selected.length, + findingIds, + shadowFindingIds: findingIds, + stale: served.some((finding) => finding.stale), + }); - record(dir, { kind: 'index', count: lines.length, tokens: spent }); + return text; +} - return `# Project wiki (${findings.length} findings, ${lines.length} listed) +function renderSessionIndex(total, findings) { + const lines = findings.map((finding) => { + const freshness = finding.stale + ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` + : ''; + return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; + }); + return `# Project wiki (${total} findings, ${findings.length} listed) Established in previous sessions. Call wiki_query with a key for detail, or just work -- findings anchored to a file are surfaced automatically when you touch it. diff --git a/integrations/codex/hooks/lib/metrics.mjs b/integrations/codex/hooks/lib/metrics.mjs index 4fe1882ed..3e4c4bb59 100644 --- a/integrations/codex/hooks/lib/metrics.mjs +++ b/integrations/codex/hooks/lib/metrics.mjs @@ -98,6 +98,8 @@ export const EVIDENCE_KINDS = new Set([ 'tool-outcome', 'episode-outcome', 'eval-run', + 'handoff-run', + 'concurrency-run', 'finding-feedback', 'retrieval-decision', 'mcp-client', @@ -1075,11 +1077,166 @@ function pairedEffects(runs, controlArm, treatmentArm, contrastType = 'increment function matchesFilters(event, filters) { for (const field of ['client', 'clientVersion', 'model', 'modelVersion', 'taskId', 'arm']) { - if (filters[field] && String(event[field] || '') !== String(filters[field])) return false; + if (!filters[field]) continue; + let actual = event[field]; + if (event.kind === 'handoff-run') { + actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + } else if (event.kind === 'concurrency-run') { + actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + } + if (String(actual || '') !== String(filters[field])) return false; } return true; } +function handoffArmMetrics(runs) { + const consumers = runs.map((run) => run.consumer || {}); + const count = (field) => consumers.filter((consumer) => consumer[field] === true).length; + const interval = (field) => bootstrapMeanInterval( + consumers.map((consumer) => consumer[field]).filter((value) => numeric(value) !== null), + { seed: `handoff:${field}` } + ); + return { + runs: runs.length, + correctness: proportionInterval(count('correct'), runs.length), + firstPass: proportionInterval(count('firstPass'), runs.length), + mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), + mistakeExecuted: proportionInterval(count('mistakeExecuted'), runs.length), + totalTokens: interval('totalTokens'), + toolCalls: interval('toolCalls'), + failedToolCalls: interval('failedToolCalls'), + latencyMs: interval('latencyMs'), + }; +} + +function handoffPairedEffect(runs, controlArm, treatmentArm) { + const pairs = new Map(); + for (const run of runs) { + if (!run.pairId || ![controlArm, treatmentArm].includes(run.arm)) continue; + if (!pairs.has(run.pairId)) pairs.set(run.pairId, {}); + pairs.get(run.pairId)[run.arm] = run; + } + const complete = [...pairs.values()].filter((pair) => pair[controlArm] && pair[treatmentArm]); + const effect = (field, positiveWhenLower = true) => bootstrapMeanInterval( + complete.map((pair) => { + const control = numeric(pair[controlArm].consumer?.[field]); + const treatment = numeric(pair[treatmentArm].consumer?.[field]); + if (control === null || treatment === null) return null; + return positiveWhenLower ? control - treatment : treatment - control; + }).filter((value) => value !== null), + { seed: `handoff:${controlArm}:${treatmentArm}:${field}` } + ); + return { + comparison: `${treatmentArm} vs ${controlArm}`, + pairs: complete.length, + attemptedMistakesPrevented: effect('mistakeAttempted'), + executedMistakesPrevented: effect('mistakeExecuted'), + firstPassDelta: effect('firstPass', false), + correctnessDelta: effect('correct', false), + totalTokensSaved: effect('totalTokens'), + toolCallsAvoided: effect('toolCalls'), + failedToolCallsAvoided: effect('failedToolCalls'), + latencyMsSaved: effect('latencyMs'), + }; +} + +const handoffKey = (run) => [ + run.producer?.client || 'unknown', run.producer?.model || 'unknown', + run.consumer?.client || 'unknown', run.consumer?.model || 'unknown', + run.scenarioId || 'unknown', +].join('|'); + +function handoffCohorts(runs) { + const grouped = new Map(); + for (const run of runs) { + const key = handoffKey(run); + if (!grouped.has(key)) grouped.set(key, []); + grouped.get(key).push(run); + } + const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + return [...grouped.entries()].map(([key, rows]) => { + const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ + arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), + ])); + const naturalRows = rows.filter((row) => row.arm === 'natural'); + const naturalVsEmpty = handoffPairedEffect(rows, 'empty', 'natural'); + const naturalVsOracle = handoffPairedEffect(rows, 'oracle', 'natural'); + const captureRate = naturalRows.length + ? naturalRows.filter((row) => row.producer?.captureSuccess).length / naturalRows.length + : null; + const preActionDeliveryRate = naturalRows.length + ? naturalRows.filter((row) => row.delivery?.beforeFirstExecutedMistake).length / naturalRows.length + : null; + const emptyRecurrence = arms.empty.mistakeExecuted.rate; + const naturalRecurrence = arms.natural.mistakeExecuted.rate; + const relativeRecurrenceReduction = emptyRecurrence + ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence + : null; + const controlsSafe = ['irrelevant', 'stale'].every((arm) => + arms[arm].correctness.rate === null + || arms.empty.correctness.rate === null + || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + ); + const gates = { + minimumPairs: naturalVsEmpty.pairs >= minimumPairs, + capture: captureRate !== null && captureRate >= 0.8, + recurrenceMagnitude: + relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, + recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, + correctness: + (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, + negativeControls: controlsSafe, + }; + const claimReady = Object.values(gates).every(Boolean); + return { + key, + producerClient: rows[0]?.producer?.client || 'unknown', + producerModel: rows[0]?.producer?.model || null, + consumerClient: rows[0]?.consumer?.client || 'unknown', + consumerModel: rows[0]?.consumer?.model || null, + scenarioId: rows[0]?.scenarioId || null, + arms, + effects: { naturalVsEmpty, naturalVsOracle }, + captureRate, + preActionDeliveryRate, + relativeRecurrenceReduction, + gates, + evidenceStatus: claimReady + ? 'pre-registered transfer gates passed' + : `insufficient or failed transfer gates (need ${minimumPairs} pairs)`, + }; + }); +} + +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; + +function concurrencySummary(runs) { + const natural = runs.filter((run) => run.arm === 'natural'); + const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); + const captures = natural.reduce((sum, run) => sum + (run.captureSuccesses || 0), 0); + const integrityPasses = natural.filter((run) => + run.integrity?.zeroLoss + && run.integrity?.parseable + && run.integrity?.orphanedFindings === 0 + ).length; + const delivered = natural.reduce((sum, run) => sum + (run.delivery?.delivered || 0), 0); + const expected = natural.reduce((sum, run) => sum + (run.delivery?.expected || 0), 0); + const effect = handoffPairedEffect(runs, 'empty', 'natural'); + return { + runs: runs.length, + naturalRuns: natural.length, + writers, + captureRate: writers ? captures / writers : null, + integrityPassRate: natural.length ? integrityPasses / natural.length : null, + deliveryCoverage: expected ? delivered / expected : null, + naturalCorrectness: proportionInterval( + natural.filter((run) => run.consumer?.correct).length, natural.length + ), + effect, + }; +} + /** * The dashboard-facing evidence console. Deterministic fixture verification, * live hook traces, and randomized model evals remain visibly distinct: only @@ -1088,6 +1245,8 @@ function matchesFilters(event, filters) { export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { const evidence = readEvidence(dir).filter((event) => matchesFilters(event, filters)); const runs = evidence.filter((event) => event.kind === 'eval-run'); + const handoffRuns = evidence.filter((event) => event.kind === 'handoff-run'); + const concurrencyRuns = evidence.filter((event) => event.kind === 'concurrency-run'); const injections = evidence.filter((event) => event.kind === 'inject'); const outcomes = evidence.filter((event) => event.kind === 'tool-outcome'); const feedback = evidence.filter((event) => event.kind === 'finding-feedback'); @@ -1157,8 +1316,10 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { byInjection.has(injection.injectionId) ).length; const harmful = feedback.filter((event) => event.rating === 'harmful').length; + const transferCohorts = handoffCohorts(handoffRuns); + const concurrency = concurrencySummary(concurrencyRuns); return { - schemaVersion: 2, + schemaVersion: 3, generatedAt: new Date().toISOString(), filters, summary: { @@ -1166,6 +1327,8 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { joinedOutcomes: joined, causalJoinCoverage: injections.length ? joined / injections.length : null, evalRuns: runs.length, + handoffRuns: handoffRuns.length, + concurrencyRuns: concurrencyRuns.length, cohorts: cohorts.length, harmfulFeedback: harmful, harmRate: feedback.length ? harmful / feedback.length : null, @@ -1174,12 +1337,16 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { : 'insufficient randomized evidence', }, cohorts, + transferCohorts, + concurrency, episodes: traced, methodology: { intervals: 'deterministic percentile bootstrap (95%); Wilson interval for correctness', causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, + handoffMinimumPairs: + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/codex/plugin/hooks/lib/adapter.mjs b/integrations/codex/plugin/hooks/lib/adapter.mjs index 2bb0082b9..c99d512cc 100644 --- a/integrations/codex/plugin/hooks/lib/adapter.mjs +++ b/integrations/codex/plugin/hooks/lib/adapter.mjs @@ -52,6 +52,8 @@ import { forSharedCommand, forTouch, noteActClasses, + sessionIndex, + standingRules, } from './inject.mjs'; import { indexFile } from './staleness.mjs'; import { isArchived } from './transcript.mjs'; @@ -420,7 +422,30 @@ export async function run(clientName, event) { if (event === 'session-start') { if (!features.routing && !features.retrieval && !features.harvest) process.exit(0); - const output = contextOutput(client, eventName, policyText(client.canDeny)); + // Some hook hosts invoke SessionStart without closing stdin. Waiting the + // full pre-tool timeout would turn optional context into a five-second + // startup tax; lifecycle payloads are tiny and arrive immediately when + // the host supplies one. + const raw = await readStdin({ timeoutMs: 250 }) || {}; + const parts = [policyText(client.canDeny)]; + if (features.retrieval) { + try { + const cwd = raw.cwd || raw.working_directory || process.cwd(); + const dir = wikiDir(projectRootFor(join(cwd, '__session__'), cwd)); + // SessionStart needs hashes and claims, not stored file bodies. Parsing + // the snapshot sidecar here would make startup scale with repository + // history even though this compact index never renders a diff. + const graph = load(dir); + const episode = episodeMeta({ client: clientName, raw }); + const rules = standingRules(dir, graph); + if (rules) parts.push(rules); + const index = sessionIndex(dir, graph, { episode }); + if (index) parts.push(index); + } catch { + // Retrieval is optional context; the policy must still reach the model. + } + } + const output = contextOutput(client, eventName, parts.join('\n\n')); if (output) emit(output); process.exit(0); } diff --git a/integrations/codex/plugin/hooks/lib/inject.mjs b/integrations/codex/plugin/hooks/lib/inject.mjs index 993752c44..db1ecec4a 100644 --- a/integrations/codex/plugin/hooks/lib/inject.mjs +++ b/integrations/codex/plugin/hooks/lib/inject.mjs @@ -779,13 +779,21 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph) { +export function sessionIndex(dir, graph, { episode = {} } = {}) { const budget = indexBudget(dir); // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. const findings = [...graph.nodes.values()] - .filter((n) => n.kind === 'finding' && !n.retired && typeof n.claim === 'string'); + .filter((n) => + n.kind === 'finding' + && !n.retired + && typeof n.claim === 'string' + // These are already rendered in full by `standingRules`; listing them a + // second time spends tokens without adding information. + && n.pinned !== true + && !(n.type === 'feedback' && n.origin === 'human') + ); if (!findings.length) return null; const now = Date.now(); @@ -793,20 +801,68 @@ export function sessionIndex(dir, graph) { ((b.confidence || 0.5) / (1 + (now - (b.at || now)) / 2.6e9)) - ((a.confidence || 0.5) / (1 + (now - (a.at || now)) / 2.6e9))); - const lines = []; - let spent = 0; + const selected = []; for (const finding of ranked) { - const line = `- ${finding.key}: ${finding.claim.slice(0, 90)}`; // claim guaranteed above - const cost = estimate(line); - if (spent + cost > budget) break; - lines.push(line); - spent += cost; + // Staleness checks can touch disk, so bound the candidates BEFORE serving + // them rather than checking an arbitrarily large graph on SessionStart. + const candidate = [...selected, finding]; + const preview = renderSessionIndex(findings.length, candidate); + if (estimate(preview) > budget) break; + selected.push(finding); } - if (!lines.length) return null; + if (!selected.length) return null; + + // `serve` is the only path allowed to hand a finding to a model. In + // particular, activating this previously-unwired index must not create a new + // path that labels an invalidated content claim as current. + const served = serve(graph, selected); + if (!served.length) return null; + // A stale marker is longer than the fresh preview used for candidate + // selection. Trim from the lowest-ranked end until the ACTUAL message fits. + while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + served.pop(); + } + if (!served.length) return null; + const text = renderSessionIndex(findings.length, served); + const spent = estimate(text); + const findingIds = served.map((finding) => finding.key); + + record(dir, { + ...episode, + kind: 'index', + surface: 'session-start', + count: served.length, + tokens: spent, + findingIds, + staleCount: served.filter((finding) => finding.stale).length, + }); + record(dir, { + ...episode, + kind: 'inject', + surface: 'session-start', + anchor: 'session-index', + holdout: false, + tokens: spent, + deliveredTokens: spent, + shadowTokens: spent, + count: served.length, + candidateCount: selected.length, + findingIds, + shadowFindingIds: findingIds, + stale: served.some((finding) => finding.stale), + }); - record(dir, { kind: 'index', count: lines.length, tokens: spent }); + return text; +} - return `# Project wiki (${findings.length} findings, ${lines.length} listed) +function renderSessionIndex(total, findings) { + const lines = findings.map((finding) => { + const freshness = finding.stale + ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` + : ''; + return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; + }); + return `# Project wiki (${total} findings, ${findings.length} listed) Established in previous sessions. Call wiki_query with a key for detail, or just work -- findings anchored to a file are surfaced automatically when you touch it. diff --git a/integrations/codex/plugin/hooks/lib/metrics.mjs b/integrations/codex/plugin/hooks/lib/metrics.mjs index 4fe1882ed..3e4c4bb59 100644 --- a/integrations/codex/plugin/hooks/lib/metrics.mjs +++ b/integrations/codex/plugin/hooks/lib/metrics.mjs @@ -98,6 +98,8 @@ export const EVIDENCE_KINDS = new Set([ 'tool-outcome', 'episode-outcome', 'eval-run', + 'handoff-run', + 'concurrency-run', 'finding-feedback', 'retrieval-decision', 'mcp-client', @@ -1075,11 +1077,166 @@ function pairedEffects(runs, controlArm, treatmentArm, contrastType = 'increment function matchesFilters(event, filters) { for (const field of ['client', 'clientVersion', 'model', 'modelVersion', 'taskId', 'arm']) { - if (filters[field] && String(event[field] || '') !== String(filters[field])) return false; + if (!filters[field]) continue; + let actual = event[field]; + if (event.kind === 'handoff-run') { + actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + } else if (event.kind === 'concurrency-run') { + actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + } + if (String(actual || '') !== String(filters[field])) return false; } return true; } +function handoffArmMetrics(runs) { + const consumers = runs.map((run) => run.consumer || {}); + const count = (field) => consumers.filter((consumer) => consumer[field] === true).length; + const interval = (field) => bootstrapMeanInterval( + consumers.map((consumer) => consumer[field]).filter((value) => numeric(value) !== null), + { seed: `handoff:${field}` } + ); + return { + runs: runs.length, + correctness: proportionInterval(count('correct'), runs.length), + firstPass: proportionInterval(count('firstPass'), runs.length), + mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), + mistakeExecuted: proportionInterval(count('mistakeExecuted'), runs.length), + totalTokens: interval('totalTokens'), + toolCalls: interval('toolCalls'), + failedToolCalls: interval('failedToolCalls'), + latencyMs: interval('latencyMs'), + }; +} + +function handoffPairedEffect(runs, controlArm, treatmentArm) { + const pairs = new Map(); + for (const run of runs) { + if (!run.pairId || ![controlArm, treatmentArm].includes(run.arm)) continue; + if (!pairs.has(run.pairId)) pairs.set(run.pairId, {}); + pairs.get(run.pairId)[run.arm] = run; + } + const complete = [...pairs.values()].filter((pair) => pair[controlArm] && pair[treatmentArm]); + const effect = (field, positiveWhenLower = true) => bootstrapMeanInterval( + complete.map((pair) => { + const control = numeric(pair[controlArm].consumer?.[field]); + const treatment = numeric(pair[treatmentArm].consumer?.[field]); + if (control === null || treatment === null) return null; + return positiveWhenLower ? control - treatment : treatment - control; + }).filter((value) => value !== null), + { seed: `handoff:${controlArm}:${treatmentArm}:${field}` } + ); + return { + comparison: `${treatmentArm} vs ${controlArm}`, + pairs: complete.length, + attemptedMistakesPrevented: effect('mistakeAttempted'), + executedMistakesPrevented: effect('mistakeExecuted'), + firstPassDelta: effect('firstPass', false), + correctnessDelta: effect('correct', false), + totalTokensSaved: effect('totalTokens'), + toolCallsAvoided: effect('toolCalls'), + failedToolCallsAvoided: effect('failedToolCalls'), + latencyMsSaved: effect('latencyMs'), + }; +} + +const handoffKey = (run) => [ + run.producer?.client || 'unknown', run.producer?.model || 'unknown', + run.consumer?.client || 'unknown', run.consumer?.model || 'unknown', + run.scenarioId || 'unknown', +].join('|'); + +function handoffCohorts(runs) { + const grouped = new Map(); + for (const run of runs) { + const key = handoffKey(run); + if (!grouped.has(key)) grouped.set(key, []); + grouped.get(key).push(run); + } + const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + return [...grouped.entries()].map(([key, rows]) => { + const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ + arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), + ])); + const naturalRows = rows.filter((row) => row.arm === 'natural'); + const naturalVsEmpty = handoffPairedEffect(rows, 'empty', 'natural'); + const naturalVsOracle = handoffPairedEffect(rows, 'oracle', 'natural'); + const captureRate = naturalRows.length + ? naturalRows.filter((row) => row.producer?.captureSuccess).length / naturalRows.length + : null; + const preActionDeliveryRate = naturalRows.length + ? naturalRows.filter((row) => row.delivery?.beforeFirstExecutedMistake).length / naturalRows.length + : null; + const emptyRecurrence = arms.empty.mistakeExecuted.rate; + const naturalRecurrence = arms.natural.mistakeExecuted.rate; + const relativeRecurrenceReduction = emptyRecurrence + ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence + : null; + const controlsSafe = ['irrelevant', 'stale'].every((arm) => + arms[arm].correctness.rate === null + || arms.empty.correctness.rate === null + || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + ); + const gates = { + minimumPairs: naturalVsEmpty.pairs >= minimumPairs, + capture: captureRate !== null && captureRate >= 0.8, + recurrenceMagnitude: + relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, + recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, + correctness: + (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, + negativeControls: controlsSafe, + }; + const claimReady = Object.values(gates).every(Boolean); + return { + key, + producerClient: rows[0]?.producer?.client || 'unknown', + producerModel: rows[0]?.producer?.model || null, + consumerClient: rows[0]?.consumer?.client || 'unknown', + consumerModel: rows[0]?.consumer?.model || null, + scenarioId: rows[0]?.scenarioId || null, + arms, + effects: { naturalVsEmpty, naturalVsOracle }, + captureRate, + preActionDeliveryRate, + relativeRecurrenceReduction, + gates, + evidenceStatus: claimReady + ? 'pre-registered transfer gates passed' + : `insufficient or failed transfer gates (need ${minimumPairs} pairs)`, + }; + }); +} + +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; + +function concurrencySummary(runs) { + const natural = runs.filter((run) => run.arm === 'natural'); + const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); + const captures = natural.reduce((sum, run) => sum + (run.captureSuccesses || 0), 0); + const integrityPasses = natural.filter((run) => + run.integrity?.zeroLoss + && run.integrity?.parseable + && run.integrity?.orphanedFindings === 0 + ).length; + const delivered = natural.reduce((sum, run) => sum + (run.delivery?.delivered || 0), 0); + const expected = natural.reduce((sum, run) => sum + (run.delivery?.expected || 0), 0); + const effect = handoffPairedEffect(runs, 'empty', 'natural'); + return { + runs: runs.length, + naturalRuns: natural.length, + writers, + captureRate: writers ? captures / writers : null, + integrityPassRate: natural.length ? integrityPasses / natural.length : null, + deliveryCoverage: expected ? delivered / expected : null, + naturalCorrectness: proportionInterval( + natural.filter((run) => run.consumer?.correct).length, natural.length + ), + effect, + }; +} + /** * The dashboard-facing evidence console. Deterministic fixture verification, * live hook traces, and randomized model evals remain visibly distinct: only @@ -1088,6 +1245,8 @@ function matchesFilters(event, filters) { export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { const evidence = readEvidence(dir).filter((event) => matchesFilters(event, filters)); const runs = evidence.filter((event) => event.kind === 'eval-run'); + const handoffRuns = evidence.filter((event) => event.kind === 'handoff-run'); + const concurrencyRuns = evidence.filter((event) => event.kind === 'concurrency-run'); const injections = evidence.filter((event) => event.kind === 'inject'); const outcomes = evidence.filter((event) => event.kind === 'tool-outcome'); const feedback = evidence.filter((event) => event.kind === 'finding-feedback'); @@ -1157,8 +1316,10 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { byInjection.has(injection.injectionId) ).length; const harmful = feedback.filter((event) => event.rating === 'harmful').length; + const transferCohorts = handoffCohorts(handoffRuns); + const concurrency = concurrencySummary(concurrencyRuns); return { - schemaVersion: 2, + schemaVersion: 3, generatedAt: new Date().toISOString(), filters, summary: { @@ -1166,6 +1327,8 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { joinedOutcomes: joined, causalJoinCoverage: injections.length ? joined / injections.length : null, evalRuns: runs.length, + handoffRuns: handoffRuns.length, + concurrencyRuns: concurrencyRuns.length, cohorts: cohorts.length, harmfulFeedback: harmful, harmRate: feedback.length ? harmful / feedback.length : null, @@ -1174,12 +1337,16 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { : 'insufficient randomized evidence', }, cohorts, + transferCohorts, + concurrency, episodes: traced, methodology: { intervals: 'deterministic percentile bootstrap (95%); Wilson interval for correctness', causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, + handoffMinimumPairs: + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/copilot/.github/hooks/lib/adapter.mjs b/integrations/copilot/.github/hooks/lib/adapter.mjs index 2bb0082b9..c99d512cc 100644 --- a/integrations/copilot/.github/hooks/lib/adapter.mjs +++ b/integrations/copilot/.github/hooks/lib/adapter.mjs @@ -52,6 +52,8 @@ import { forSharedCommand, forTouch, noteActClasses, + sessionIndex, + standingRules, } from './inject.mjs'; import { indexFile } from './staleness.mjs'; import { isArchived } from './transcript.mjs'; @@ -420,7 +422,30 @@ export async function run(clientName, event) { if (event === 'session-start') { if (!features.routing && !features.retrieval && !features.harvest) process.exit(0); - const output = contextOutput(client, eventName, policyText(client.canDeny)); + // Some hook hosts invoke SessionStart without closing stdin. Waiting the + // full pre-tool timeout would turn optional context into a five-second + // startup tax; lifecycle payloads are tiny and arrive immediately when + // the host supplies one. + const raw = await readStdin({ timeoutMs: 250 }) || {}; + const parts = [policyText(client.canDeny)]; + if (features.retrieval) { + try { + const cwd = raw.cwd || raw.working_directory || process.cwd(); + const dir = wikiDir(projectRootFor(join(cwd, '__session__'), cwd)); + // SessionStart needs hashes and claims, not stored file bodies. Parsing + // the snapshot sidecar here would make startup scale with repository + // history even though this compact index never renders a diff. + const graph = load(dir); + const episode = episodeMeta({ client: clientName, raw }); + const rules = standingRules(dir, graph); + if (rules) parts.push(rules); + const index = sessionIndex(dir, graph, { episode }); + if (index) parts.push(index); + } catch { + // Retrieval is optional context; the policy must still reach the model. + } + } + const output = contextOutput(client, eventName, parts.join('\n\n')); if (output) emit(output); process.exit(0); } diff --git a/integrations/copilot/.github/hooks/lib/inject.mjs b/integrations/copilot/.github/hooks/lib/inject.mjs index 993752c44..db1ecec4a 100644 --- a/integrations/copilot/.github/hooks/lib/inject.mjs +++ b/integrations/copilot/.github/hooks/lib/inject.mjs @@ -779,13 +779,21 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph) { +export function sessionIndex(dir, graph, { episode = {} } = {}) { const budget = indexBudget(dir); // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. const findings = [...graph.nodes.values()] - .filter((n) => n.kind === 'finding' && !n.retired && typeof n.claim === 'string'); + .filter((n) => + n.kind === 'finding' + && !n.retired + && typeof n.claim === 'string' + // These are already rendered in full by `standingRules`; listing them a + // second time spends tokens without adding information. + && n.pinned !== true + && !(n.type === 'feedback' && n.origin === 'human') + ); if (!findings.length) return null; const now = Date.now(); @@ -793,20 +801,68 @@ export function sessionIndex(dir, graph) { ((b.confidence || 0.5) / (1 + (now - (b.at || now)) / 2.6e9)) - ((a.confidence || 0.5) / (1 + (now - (a.at || now)) / 2.6e9))); - const lines = []; - let spent = 0; + const selected = []; for (const finding of ranked) { - const line = `- ${finding.key}: ${finding.claim.slice(0, 90)}`; // claim guaranteed above - const cost = estimate(line); - if (spent + cost > budget) break; - lines.push(line); - spent += cost; + // Staleness checks can touch disk, so bound the candidates BEFORE serving + // them rather than checking an arbitrarily large graph on SessionStart. + const candidate = [...selected, finding]; + const preview = renderSessionIndex(findings.length, candidate); + if (estimate(preview) > budget) break; + selected.push(finding); } - if (!lines.length) return null; + if (!selected.length) return null; + + // `serve` is the only path allowed to hand a finding to a model. In + // particular, activating this previously-unwired index must not create a new + // path that labels an invalidated content claim as current. + const served = serve(graph, selected); + if (!served.length) return null; + // A stale marker is longer than the fresh preview used for candidate + // selection. Trim from the lowest-ranked end until the ACTUAL message fits. + while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + served.pop(); + } + if (!served.length) return null; + const text = renderSessionIndex(findings.length, served); + const spent = estimate(text); + const findingIds = served.map((finding) => finding.key); + + record(dir, { + ...episode, + kind: 'index', + surface: 'session-start', + count: served.length, + tokens: spent, + findingIds, + staleCount: served.filter((finding) => finding.stale).length, + }); + record(dir, { + ...episode, + kind: 'inject', + surface: 'session-start', + anchor: 'session-index', + holdout: false, + tokens: spent, + deliveredTokens: spent, + shadowTokens: spent, + count: served.length, + candidateCount: selected.length, + findingIds, + shadowFindingIds: findingIds, + stale: served.some((finding) => finding.stale), + }); - record(dir, { kind: 'index', count: lines.length, tokens: spent }); + return text; +} - return `# Project wiki (${findings.length} findings, ${lines.length} listed) +function renderSessionIndex(total, findings) { + const lines = findings.map((finding) => { + const freshness = finding.stale + ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` + : ''; + return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; + }); + return `# Project wiki (${total} findings, ${findings.length} listed) Established in previous sessions. Call wiki_query with a key for detail, or just work -- findings anchored to a file are surfaced automatically when you touch it. diff --git a/integrations/copilot/.github/hooks/lib/metrics.mjs b/integrations/copilot/.github/hooks/lib/metrics.mjs index 4fe1882ed..3e4c4bb59 100644 --- a/integrations/copilot/.github/hooks/lib/metrics.mjs +++ b/integrations/copilot/.github/hooks/lib/metrics.mjs @@ -98,6 +98,8 @@ export const EVIDENCE_KINDS = new Set([ 'tool-outcome', 'episode-outcome', 'eval-run', + 'handoff-run', + 'concurrency-run', 'finding-feedback', 'retrieval-decision', 'mcp-client', @@ -1075,11 +1077,166 @@ function pairedEffects(runs, controlArm, treatmentArm, contrastType = 'increment function matchesFilters(event, filters) { for (const field of ['client', 'clientVersion', 'model', 'modelVersion', 'taskId', 'arm']) { - if (filters[field] && String(event[field] || '') !== String(filters[field])) return false; + if (!filters[field]) continue; + let actual = event[field]; + if (event.kind === 'handoff-run') { + actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + } else if (event.kind === 'concurrency-run') { + actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + } + if (String(actual || '') !== String(filters[field])) return false; } return true; } +function handoffArmMetrics(runs) { + const consumers = runs.map((run) => run.consumer || {}); + const count = (field) => consumers.filter((consumer) => consumer[field] === true).length; + const interval = (field) => bootstrapMeanInterval( + consumers.map((consumer) => consumer[field]).filter((value) => numeric(value) !== null), + { seed: `handoff:${field}` } + ); + return { + runs: runs.length, + correctness: proportionInterval(count('correct'), runs.length), + firstPass: proportionInterval(count('firstPass'), runs.length), + mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), + mistakeExecuted: proportionInterval(count('mistakeExecuted'), runs.length), + totalTokens: interval('totalTokens'), + toolCalls: interval('toolCalls'), + failedToolCalls: interval('failedToolCalls'), + latencyMs: interval('latencyMs'), + }; +} + +function handoffPairedEffect(runs, controlArm, treatmentArm) { + const pairs = new Map(); + for (const run of runs) { + if (!run.pairId || ![controlArm, treatmentArm].includes(run.arm)) continue; + if (!pairs.has(run.pairId)) pairs.set(run.pairId, {}); + pairs.get(run.pairId)[run.arm] = run; + } + const complete = [...pairs.values()].filter((pair) => pair[controlArm] && pair[treatmentArm]); + const effect = (field, positiveWhenLower = true) => bootstrapMeanInterval( + complete.map((pair) => { + const control = numeric(pair[controlArm].consumer?.[field]); + const treatment = numeric(pair[treatmentArm].consumer?.[field]); + if (control === null || treatment === null) return null; + return positiveWhenLower ? control - treatment : treatment - control; + }).filter((value) => value !== null), + { seed: `handoff:${controlArm}:${treatmentArm}:${field}` } + ); + return { + comparison: `${treatmentArm} vs ${controlArm}`, + pairs: complete.length, + attemptedMistakesPrevented: effect('mistakeAttempted'), + executedMistakesPrevented: effect('mistakeExecuted'), + firstPassDelta: effect('firstPass', false), + correctnessDelta: effect('correct', false), + totalTokensSaved: effect('totalTokens'), + toolCallsAvoided: effect('toolCalls'), + failedToolCallsAvoided: effect('failedToolCalls'), + latencyMsSaved: effect('latencyMs'), + }; +} + +const handoffKey = (run) => [ + run.producer?.client || 'unknown', run.producer?.model || 'unknown', + run.consumer?.client || 'unknown', run.consumer?.model || 'unknown', + run.scenarioId || 'unknown', +].join('|'); + +function handoffCohorts(runs) { + const grouped = new Map(); + for (const run of runs) { + const key = handoffKey(run); + if (!grouped.has(key)) grouped.set(key, []); + grouped.get(key).push(run); + } + const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + return [...grouped.entries()].map(([key, rows]) => { + const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ + arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), + ])); + const naturalRows = rows.filter((row) => row.arm === 'natural'); + const naturalVsEmpty = handoffPairedEffect(rows, 'empty', 'natural'); + const naturalVsOracle = handoffPairedEffect(rows, 'oracle', 'natural'); + const captureRate = naturalRows.length + ? naturalRows.filter((row) => row.producer?.captureSuccess).length / naturalRows.length + : null; + const preActionDeliveryRate = naturalRows.length + ? naturalRows.filter((row) => row.delivery?.beforeFirstExecutedMistake).length / naturalRows.length + : null; + const emptyRecurrence = arms.empty.mistakeExecuted.rate; + const naturalRecurrence = arms.natural.mistakeExecuted.rate; + const relativeRecurrenceReduction = emptyRecurrence + ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence + : null; + const controlsSafe = ['irrelevant', 'stale'].every((arm) => + arms[arm].correctness.rate === null + || arms.empty.correctness.rate === null + || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + ); + const gates = { + minimumPairs: naturalVsEmpty.pairs >= minimumPairs, + capture: captureRate !== null && captureRate >= 0.8, + recurrenceMagnitude: + relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, + recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, + correctness: + (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, + negativeControls: controlsSafe, + }; + const claimReady = Object.values(gates).every(Boolean); + return { + key, + producerClient: rows[0]?.producer?.client || 'unknown', + producerModel: rows[0]?.producer?.model || null, + consumerClient: rows[0]?.consumer?.client || 'unknown', + consumerModel: rows[0]?.consumer?.model || null, + scenarioId: rows[0]?.scenarioId || null, + arms, + effects: { naturalVsEmpty, naturalVsOracle }, + captureRate, + preActionDeliveryRate, + relativeRecurrenceReduction, + gates, + evidenceStatus: claimReady + ? 'pre-registered transfer gates passed' + : `insufficient or failed transfer gates (need ${minimumPairs} pairs)`, + }; + }); +} + +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; + +function concurrencySummary(runs) { + const natural = runs.filter((run) => run.arm === 'natural'); + const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); + const captures = natural.reduce((sum, run) => sum + (run.captureSuccesses || 0), 0); + const integrityPasses = natural.filter((run) => + run.integrity?.zeroLoss + && run.integrity?.parseable + && run.integrity?.orphanedFindings === 0 + ).length; + const delivered = natural.reduce((sum, run) => sum + (run.delivery?.delivered || 0), 0); + const expected = natural.reduce((sum, run) => sum + (run.delivery?.expected || 0), 0); + const effect = handoffPairedEffect(runs, 'empty', 'natural'); + return { + runs: runs.length, + naturalRuns: natural.length, + writers, + captureRate: writers ? captures / writers : null, + integrityPassRate: natural.length ? integrityPasses / natural.length : null, + deliveryCoverage: expected ? delivered / expected : null, + naturalCorrectness: proportionInterval( + natural.filter((run) => run.consumer?.correct).length, natural.length + ), + effect, + }; +} + /** * The dashboard-facing evidence console. Deterministic fixture verification, * live hook traces, and randomized model evals remain visibly distinct: only @@ -1088,6 +1245,8 @@ function matchesFilters(event, filters) { export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { const evidence = readEvidence(dir).filter((event) => matchesFilters(event, filters)); const runs = evidence.filter((event) => event.kind === 'eval-run'); + const handoffRuns = evidence.filter((event) => event.kind === 'handoff-run'); + const concurrencyRuns = evidence.filter((event) => event.kind === 'concurrency-run'); const injections = evidence.filter((event) => event.kind === 'inject'); const outcomes = evidence.filter((event) => event.kind === 'tool-outcome'); const feedback = evidence.filter((event) => event.kind === 'finding-feedback'); @@ -1157,8 +1316,10 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { byInjection.has(injection.injectionId) ).length; const harmful = feedback.filter((event) => event.rating === 'harmful').length; + const transferCohorts = handoffCohorts(handoffRuns); + const concurrency = concurrencySummary(concurrencyRuns); return { - schemaVersion: 2, + schemaVersion: 3, generatedAt: new Date().toISOString(), filters, summary: { @@ -1166,6 +1327,8 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { joinedOutcomes: joined, causalJoinCoverage: injections.length ? joined / injections.length : null, evalRuns: runs.length, + handoffRuns: handoffRuns.length, + concurrencyRuns: concurrencyRuns.length, cohorts: cohorts.length, harmfulFeedback: harmful, harmRate: feedback.length ? harmful / feedback.length : null, @@ -1174,12 +1337,16 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { : 'insufficient randomized evidence', }, cohorts, + transferCohorts, + concurrency, episodes: traced, methodology: { intervals: 'deterministic percentile bootstrap (95%); Wilson interval for correctness', causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, + handoffMinimumPairs: + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/cursor/hooks/lib/adapter.mjs b/integrations/cursor/hooks/lib/adapter.mjs index 2bb0082b9..c99d512cc 100644 --- a/integrations/cursor/hooks/lib/adapter.mjs +++ b/integrations/cursor/hooks/lib/adapter.mjs @@ -52,6 +52,8 @@ import { forSharedCommand, forTouch, noteActClasses, + sessionIndex, + standingRules, } from './inject.mjs'; import { indexFile } from './staleness.mjs'; import { isArchived } from './transcript.mjs'; @@ -420,7 +422,30 @@ export async function run(clientName, event) { if (event === 'session-start') { if (!features.routing && !features.retrieval && !features.harvest) process.exit(0); - const output = contextOutput(client, eventName, policyText(client.canDeny)); + // Some hook hosts invoke SessionStart without closing stdin. Waiting the + // full pre-tool timeout would turn optional context into a five-second + // startup tax; lifecycle payloads are tiny and arrive immediately when + // the host supplies one. + const raw = await readStdin({ timeoutMs: 250 }) || {}; + const parts = [policyText(client.canDeny)]; + if (features.retrieval) { + try { + const cwd = raw.cwd || raw.working_directory || process.cwd(); + const dir = wikiDir(projectRootFor(join(cwd, '__session__'), cwd)); + // SessionStart needs hashes and claims, not stored file bodies. Parsing + // the snapshot sidecar here would make startup scale with repository + // history even though this compact index never renders a diff. + const graph = load(dir); + const episode = episodeMeta({ client: clientName, raw }); + const rules = standingRules(dir, graph); + if (rules) parts.push(rules); + const index = sessionIndex(dir, graph, { episode }); + if (index) parts.push(index); + } catch { + // Retrieval is optional context; the policy must still reach the model. + } + } + const output = contextOutput(client, eventName, parts.join('\n\n')); if (output) emit(output); process.exit(0); } diff --git a/integrations/cursor/hooks/lib/inject.mjs b/integrations/cursor/hooks/lib/inject.mjs index 993752c44..db1ecec4a 100644 --- a/integrations/cursor/hooks/lib/inject.mjs +++ b/integrations/cursor/hooks/lib/inject.mjs @@ -779,13 +779,21 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph) { +export function sessionIndex(dir, graph, { episode = {} } = {}) { const budget = indexBudget(dir); // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. const findings = [...graph.nodes.values()] - .filter((n) => n.kind === 'finding' && !n.retired && typeof n.claim === 'string'); + .filter((n) => + n.kind === 'finding' + && !n.retired + && typeof n.claim === 'string' + // These are already rendered in full by `standingRules`; listing them a + // second time spends tokens without adding information. + && n.pinned !== true + && !(n.type === 'feedback' && n.origin === 'human') + ); if (!findings.length) return null; const now = Date.now(); @@ -793,20 +801,68 @@ export function sessionIndex(dir, graph) { ((b.confidence || 0.5) / (1 + (now - (b.at || now)) / 2.6e9)) - ((a.confidence || 0.5) / (1 + (now - (a.at || now)) / 2.6e9))); - const lines = []; - let spent = 0; + const selected = []; for (const finding of ranked) { - const line = `- ${finding.key}: ${finding.claim.slice(0, 90)}`; // claim guaranteed above - const cost = estimate(line); - if (spent + cost > budget) break; - lines.push(line); - spent += cost; + // Staleness checks can touch disk, so bound the candidates BEFORE serving + // them rather than checking an arbitrarily large graph on SessionStart. + const candidate = [...selected, finding]; + const preview = renderSessionIndex(findings.length, candidate); + if (estimate(preview) > budget) break; + selected.push(finding); } - if (!lines.length) return null; + if (!selected.length) return null; + + // `serve` is the only path allowed to hand a finding to a model. In + // particular, activating this previously-unwired index must not create a new + // path that labels an invalidated content claim as current. + const served = serve(graph, selected); + if (!served.length) return null; + // A stale marker is longer than the fresh preview used for candidate + // selection. Trim from the lowest-ranked end until the ACTUAL message fits. + while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + served.pop(); + } + if (!served.length) return null; + const text = renderSessionIndex(findings.length, served); + const spent = estimate(text); + const findingIds = served.map((finding) => finding.key); + + record(dir, { + ...episode, + kind: 'index', + surface: 'session-start', + count: served.length, + tokens: spent, + findingIds, + staleCount: served.filter((finding) => finding.stale).length, + }); + record(dir, { + ...episode, + kind: 'inject', + surface: 'session-start', + anchor: 'session-index', + holdout: false, + tokens: spent, + deliveredTokens: spent, + shadowTokens: spent, + count: served.length, + candidateCount: selected.length, + findingIds, + shadowFindingIds: findingIds, + stale: served.some((finding) => finding.stale), + }); - record(dir, { kind: 'index', count: lines.length, tokens: spent }); + return text; +} - return `# Project wiki (${findings.length} findings, ${lines.length} listed) +function renderSessionIndex(total, findings) { + const lines = findings.map((finding) => { + const freshness = finding.stale + ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` + : ''; + return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; + }); + return `# Project wiki (${total} findings, ${findings.length} listed) Established in previous sessions. Call wiki_query with a key for detail, or just work -- findings anchored to a file are surfaced automatically when you touch it. diff --git a/integrations/cursor/hooks/lib/metrics.mjs b/integrations/cursor/hooks/lib/metrics.mjs index 4fe1882ed..3e4c4bb59 100644 --- a/integrations/cursor/hooks/lib/metrics.mjs +++ b/integrations/cursor/hooks/lib/metrics.mjs @@ -98,6 +98,8 @@ export const EVIDENCE_KINDS = new Set([ 'tool-outcome', 'episode-outcome', 'eval-run', + 'handoff-run', + 'concurrency-run', 'finding-feedback', 'retrieval-decision', 'mcp-client', @@ -1075,11 +1077,166 @@ function pairedEffects(runs, controlArm, treatmentArm, contrastType = 'increment function matchesFilters(event, filters) { for (const field of ['client', 'clientVersion', 'model', 'modelVersion', 'taskId', 'arm']) { - if (filters[field] && String(event[field] || '') !== String(filters[field])) return false; + if (!filters[field]) continue; + let actual = event[field]; + if (event.kind === 'handoff-run') { + actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + } else if (event.kind === 'concurrency-run') { + actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + } + if (String(actual || '') !== String(filters[field])) return false; } return true; } +function handoffArmMetrics(runs) { + const consumers = runs.map((run) => run.consumer || {}); + const count = (field) => consumers.filter((consumer) => consumer[field] === true).length; + const interval = (field) => bootstrapMeanInterval( + consumers.map((consumer) => consumer[field]).filter((value) => numeric(value) !== null), + { seed: `handoff:${field}` } + ); + return { + runs: runs.length, + correctness: proportionInterval(count('correct'), runs.length), + firstPass: proportionInterval(count('firstPass'), runs.length), + mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), + mistakeExecuted: proportionInterval(count('mistakeExecuted'), runs.length), + totalTokens: interval('totalTokens'), + toolCalls: interval('toolCalls'), + failedToolCalls: interval('failedToolCalls'), + latencyMs: interval('latencyMs'), + }; +} + +function handoffPairedEffect(runs, controlArm, treatmentArm) { + const pairs = new Map(); + for (const run of runs) { + if (!run.pairId || ![controlArm, treatmentArm].includes(run.arm)) continue; + if (!pairs.has(run.pairId)) pairs.set(run.pairId, {}); + pairs.get(run.pairId)[run.arm] = run; + } + const complete = [...pairs.values()].filter((pair) => pair[controlArm] && pair[treatmentArm]); + const effect = (field, positiveWhenLower = true) => bootstrapMeanInterval( + complete.map((pair) => { + const control = numeric(pair[controlArm].consumer?.[field]); + const treatment = numeric(pair[treatmentArm].consumer?.[field]); + if (control === null || treatment === null) return null; + return positiveWhenLower ? control - treatment : treatment - control; + }).filter((value) => value !== null), + { seed: `handoff:${controlArm}:${treatmentArm}:${field}` } + ); + return { + comparison: `${treatmentArm} vs ${controlArm}`, + pairs: complete.length, + attemptedMistakesPrevented: effect('mistakeAttempted'), + executedMistakesPrevented: effect('mistakeExecuted'), + firstPassDelta: effect('firstPass', false), + correctnessDelta: effect('correct', false), + totalTokensSaved: effect('totalTokens'), + toolCallsAvoided: effect('toolCalls'), + failedToolCallsAvoided: effect('failedToolCalls'), + latencyMsSaved: effect('latencyMs'), + }; +} + +const handoffKey = (run) => [ + run.producer?.client || 'unknown', run.producer?.model || 'unknown', + run.consumer?.client || 'unknown', run.consumer?.model || 'unknown', + run.scenarioId || 'unknown', +].join('|'); + +function handoffCohorts(runs) { + const grouped = new Map(); + for (const run of runs) { + const key = handoffKey(run); + if (!grouped.has(key)) grouped.set(key, []); + grouped.get(key).push(run); + } + const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + return [...grouped.entries()].map(([key, rows]) => { + const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ + arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), + ])); + const naturalRows = rows.filter((row) => row.arm === 'natural'); + const naturalVsEmpty = handoffPairedEffect(rows, 'empty', 'natural'); + const naturalVsOracle = handoffPairedEffect(rows, 'oracle', 'natural'); + const captureRate = naturalRows.length + ? naturalRows.filter((row) => row.producer?.captureSuccess).length / naturalRows.length + : null; + const preActionDeliveryRate = naturalRows.length + ? naturalRows.filter((row) => row.delivery?.beforeFirstExecutedMistake).length / naturalRows.length + : null; + const emptyRecurrence = arms.empty.mistakeExecuted.rate; + const naturalRecurrence = arms.natural.mistakeExecuted.rate; + const relativeRecurrenceReduction = emptyRecurrence + ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence + : null; + const controlsSafe = ['irrelevant', 'stale'].every((arm) => + arms[arm].correctness.rate === null + || arms.empty.correctness.rate === null + || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + ); + const gates = { + minimumPairs: naturalVsEmpty.pairs >= minimumPairs, + capture: captureRate !== null && captureRate >= 0.8, + recurrenceMagnitude: + relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, + recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, + correctness: + (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, + negativeControls: controlsSafe, + }; + const claimReady = Object.values(gates).every(Boolean); + return { + key, + producerClient: rows[0]?.producer?.client || 'unknown', + producerModel: rows[0]?.producer?.model || null, + consumerClient: rows[0]?.consumer?.client || 'unknown', + consumerModel: rows[0]?.consumer?.model || null, + scenarioId: rows[0]?.scenarioId || null, + arms, + effects: { naturalVsEmpty, naturalVsOracle }, + captureRate, + preActionDeliveryRate, + relativeRecurrenceReduction, + gates, + evidenceStatus: claimReady + ? 'pre-registered transfer gates passed' + : `insufficient or failed transfer gates (need ${minimumPairs} pairs)`, + }; + }); +} + +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; + +function concurrencySummary(runs) { + const natural = runs.filter((run) => run.arm === 'natural'); + const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); + const captures = natural.reduce((sum, run) => sum + (run.captureSuccesses || 0), 0); + const integrityPasses = natural.filter((run) => + run.integrity?.zeroLoss + && run.integrity?.parseable + && run.integrity?.orphanedFindings === 0 + ).length; + const delivered = natural.reduce((sum, run) => sum + (run.delivery?.delivered || 0), 0); + const expected = natural.reduce((sum, run) => sum + (run.delivery?.expected || 0), 0); + const effect = handoffPairedEffect(runs, 'empty', 'natural'); + return { + runs: runs.length, + naturalRuns: natural.length, + writers, + captureRate: writers ? captures / writers : null, + integrityPassRate: natural.length ? integrityPasses / natural.length : null, + deliveryCoverage: expected ? delivered / expected : null, + naturalCorrectness: proportionInterval( + natural.filter((run) => run.consumer?.correct).length, natural.length + ), + effect, + }; +} + /** * The dashboard-facing evidence console. Deterministic fixture verification, * live hook traces, and randomized model evals remain visibly distinct: only @@ -1088,6 +1245,8 @@ function matchesFilters(event, filters) { export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { const evidence = readEvidence(dir).filter((event) => matchesFilters(event, filters)); const runs = evidence.filter((event) => event.kind === 'eval-run'); + const handoffRuns = evidence.filter((event) => event.kind === 'handoff-run'); + const concurrencyRuns = evidence.filter((event) => event.kind === 'concurrency-run'); const injections = evidence.filter((event) => event.kind === 'inject'); const outcomes = evidence.filter((event) => event.kind === 'tool-outcome'); const feedback = evidence.filter((event) => event.kind === 'finding-feedback'); @@ -1157,8 +1316,10 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { byInjection.has(injection.injectionId) ).length; const harmful = feedback.filter((event) => event.rating === 'harmful').length; + const transferCohorts = handoffCohorts(handoffRuns); + const concurrency = concurrencySummary(concurrencyRuns); return { - schemaVersion: 2, + schemaVersion: 3, generatedAt: new Date().toISOString(), filters, summary: { @@ -1166,6 +1327,8 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { joinedOutcomes: joined, causalJoinCoverage: injections.length ? joined / injections.length : null, evalRuns: runs.length, + handoffRuns: handoffRuns.length, + concurrencyRuns: concurrencyRuns.length, cohorts: cohorts.length, harmfulFeedback: harmful, harmRate: feedback.length ? harmful / feedback.length : null, @@ -1174,12 +1337,16 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { : 'insufficient randomized evidence', }, cohorts, + transferCohorts, + concurrency, episodes: traced, methodology: { intervals: 'deterministic percentile bootstrap (95%); Wilson interval for correctness', causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, + handoffMinimumPairs: + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/gemini/hooks/lib/adapter.mjs b/integrations/gemini/hooks/lib/adapter.mjs index 2bb0082b9..c99d512cc 100644 --- a/integrations/gemini/hooks/lib/adapter.mjs +++ b/integrations/gemini/hooks/lib/adapter.mjs @@ -52,6 +52,8 @@ import { forSharedCommand, forTouch, noteActClasses, + sessionIndex, + standingRules, } from './inject.mjs'; import { indexFile } from './staleness.mjs'; import { isArchived } from './transcript.mjs'; @@ -420,7 +422,30 @@ export async function run(clientName, event) { if (event === 'session-start') { if (!features.routing && !features.retrieval && !features.harvest) process.exit(0); - const output = contextOutput(client, eventName, policyText(client.canDeny)); + // Some hook hosts invoke SessionStart without closing stdin. Waiting the + // full pre-tool timeout would turn optional context into a five-second + // startup tax; lifecycle payloads are tiny and arrive immediately when + // the host supplies one. + const raw = await readStdin({ timeoutMs: 250 }) || {}; + const parts = [policyText(client.canDeny)]; + if (features.retrieval) { + try { + const cwd = raw.cwd || raw.working_directory || process.cwd(); + const dir = wikiDir(projectRootFor(join(cwd, '__session__'), cwd)); + // SessionStart needs hashes and claims, not stored file bodies. Parsing + // the snapshot sidecar here would make startup scale with repository + // history even though this compact index never renders a diff. + const graph = load(dir); + const episode = episodeMeta({ client: clientName, raw }); + const rules = standingRules(dir, graph); + if (rules) parts.push(rules); + const index = sessionIndex(dir, graph, { episode }); + if (index) parts.push(index); + } catch { + // Retrieval is optional context; the policy must still reach the model. + } + } + const output = contextOutput(client, eventName, parts.join('\n\n')); if (output) emit(output); process.exit(0); } diff --git a/integrations/gemini/hooks/lib/inject.mjs b/integrations/gemini/hooks/lib/inject.mjs index 993752c44..db1ecec4a 100644 --- a/integrations/gemini/hooks/lib/inject.mjs +++ b/integrations/gemini/hooks/lib/inject.mjs @@ -779,13 +779,21 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph) { +export function sessionIndex(dir, graph, { episode = {} } = {}) { const budget = indexBudget(dir); // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. const findings = [...graph.nodes.values()] - .filter((n) => n.kind === 'finding' && !n.retired && typeof n.claim === 'string'); + .filter((n) => + n.kind === 'finding' + && !n.retired + && typeof n.claim === 'string' + // These are already rendered in full by `standingRules`; listing them a + // second time spends tokens without adding information. + && n.pinned !== true + && !(n.type === 'feedback' && n.origin === 'human') + ); if (!findings.length) return null; const now = Date.now(); @@ -793,20 +801,68 @@ export function sessionIndex(dir, graph) { ((b.confidence || 0.5) / (1 + (now - (b.at || now)) / 2.6e9)) - ((a.confidence || 0.5) / (1 + (now - (a.at || now)) / 2.6e9))); - const lines = []; - let spent = 0; + const selected = []; for (const finding of ranked) { - const line = `- ${finding.key}: ${finding.claim.slice(0, 90)}`; // claim guaranteed above - const cost = estimate(line); - if (spent + cost > budget) break; - lines.push(line); - spent += cost; + // Staleness checks can touch disk, so bound the candidates BEFORE serving + // them rather than checking an arbitrarily large graph on SessionStart. + const candidate = [...selected, finding]; + const preview = renderSessionIndex(findings.length, candidate); + if (estimate(preview) > budget) break; + selected.push(finding); } - if (!lines.length) return null; + if (!selected.length) return null; + + // `serve` is the only path allowed to hand a finding to a model. In + // particular, activating this previously-unwired index must not create a new + // path that labels an invalidated content claim as current. + const served = serve(graph, selected); + if (!served.length) return null; + // A stale marker is longer than the fresh preview used for candidate + // selection. Trim from the lowest-ranked end until the ACTUAL message fits. + while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + served.pop(); + } + if (!served.length) return null; + const text = renderSessionIndex(findings.length, served); + const spent = estimate(text); + const findingIds = served.map((finding) => finding.key); + + record(dir, { + ...episode, + kind: 'index', + surface: 'session-start', + count: served.length, + tokens: spent, + findingIds, + staleCount: served.filter((finding) => finding.stale).length, + }); + record(dir, { + ...episode, + kind: 'inject', + surface: 'session-start', + anchor: 'session-index', + holdout: false, + tokens: spent, + deliveredTokens: spent, + shadowTokens: spent, + count: served.length, + candidateCount: selected.length, + findingIds, + shadowFindingIds: findingIds, + stale: served.some((finding) => finding.stale), + }); - record(dir, { kind: 'index', count: lines.length, tokens: spent }); + return text; +} - return `# Project wiki (${findings.length} findings, ${lines.length} listed) +function renderSessionIndex(total, findings) { + const lines = findings.map((finding) => { + const freshness = finding.stale + ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` + : ''; + return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; + }); + return `# Project wiki (${total} findings, ${findings.length} listed) Established in previous sessions. Call wiki_query with a key for detail, or just work -- findings anchored to a file are surfaced automatically when you touch it. diff --git a/integrations/gemini/hooks/lib/metrics.mjs b/integrations/gemini/hooks/lib/metrics.mjs index 4fe1882ed..3e4c4bb59 100644 --- a/integrations/gemini/hooks/lib/metrics.mjs +++ b/integrations/gemini/hooks/lib/metrics.mjs @@ -98,6 +98,8 @@ export const EVIDENCE_KINDS = new Set([ 'tool-outcome', 'episode-outcome', 'eval-run', + 'handoff-run', + 'concurrency-run', 'finding-feedback', 'retrieval-decision', 'mcp-client', @@ -1075,11 +1077,166 @@ function pairedEffects(runs, controlArm, treatmentArm, contrastType = 'increment function matchesFilters(event, filters) { for (const field of ['client', 'clientVersion', 'model', 'modelVersion', 'taskId', 'arm']) { - if (filters[field] && String(event[field] || '') !== String(filters[field])) return false; + if (!filters[field]) continue; + let actual = event[field]; + if (event.kind === 'handoff-run') { + actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + } else if (event.kind === 'concurrency-run') { + actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + } + if (String(actual || '') !== String(filters[field])) return false; } return true; } +function handoffArmMetrics(runs) { + const consumers = runs.map((run) => run.consumer || {}); + const count = (field) => consumers.filter((consumer) => consumer[field] === true).length; + const interval = (field) => bootstrapMeanInterval( + consumers.map((consumer) => consumer[field]).filter((value) => numeric(value) !== null), + { seed: `handoff:${field}` } + ); + return { + runs: runs.length, + correctness: proportionInterval(count('correct'), runs.length), + firstPass: proportionInterval(count('firstPass'), runs.length), + mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), + mistakeExecuted: proportionInterval(count('mistakeExecuted'), runs.length), + totalTokens: interval('totalTokens'), + toolCalls: interval('toolCalls'), + failedToolCalls: interval('failedToolCalls'), + latencyMs: interval('latencyMs'), + }; +} + +function handoffPairedEffect(runs, controlArm, treatmentArm) { + const pairs = new Map(); + for (const run of runs) { + if (!run.pairId || ![controlArm, treatmentArm].includes(run.arm)) continue; + if (!pairs.has(run.pairId)) pairs.set(run.pairId, {}); + pairs.get(run.pairId)[run.arm] = run; + } + const complete = [...pairs.values()].filter((pair) => pair[controlArm] && pair[treatmentArm]); + const effect = (field, positiveWhenLower = true) => bootstrapMeanInterval( + complete.map((pair) => { + const control = numeric(pair[controlArm].consumer?.[field]); + const treatment = numeric(pair[treatmentArm].consumer?.[field]); + if (control === null || treatment === null) return null; + return positiveWhenLower ? control - treatment : treatment - control; + }).filter((value) => value !== null), + { seed: `handoff:${controlArm}:${treatmentArm}:${field}` } + ); + return { + comparison: `${treatmentArm} vs ${controlArm}`, + pairs: complete.length, + attemptedMistakesPrevented: effect('mistakeAttempted'), + executedMistakesPrevented: effect('mistakeExecuted'), + firstPassDelta: effect('firstPass', false), + correctnessDelta: effect('correct', false), + totalTokensSaved: effect('totalTokens'), + toolCallsAvoided: effect('toolCalls'), + failedToolCallsAvoided: effect('failedToolCalls'), + latencyMsSaved: effect('latencyMs'), + }; +} + +const handoffKey = (run) => [ + run.producer?.client || 'unknown', run.producer?.model || 'unknown', + run.consumer?.client || 'unknown', run.consumer?.model || 'unknown', + run.scenarioId || 'unknown', +].join('|'); + +function handoffCohorts(runs) { + const grouped = new Map(); + for (const run of runs) { + const key = handoffKey(run); + if (!grouped.has(key)) grouped.set(key, []); + grouped.get(key).push(run); + } + const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + return [...grouped.entries()].map(([key, rows]) => { + const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ + arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), + ])); + const naturalRows = rows.filter((row) => row.arm === 'natural'); + const naturalVsEmpty = handoffPairedEffect(rows, 'empty', 'natural'); + const naturalVsOracle = handoffPairedEffect(rows, 'oracle', 'natural'); + const captureRate = naturalRows.length + ? naturalRows.filter((row) => row.producer?.captureSuccess).length / naturalRows.length + : null; + const preActionDeliveryRate = naturalRows.length + ? naturalRows.filter((row) => row.delivery?.beforeFirstExecutedMistake).length / naturalRows.length + : null; + const emptyRecurrence = arms.empty.mistakeExecuted.rate; + const naturalRecurrence = arms.natural.mistakeExecuted.rate; + const relativeRecurrenceReduction = emptyRecurrence + ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence + : null; + const controlsSafe = ['irrelevant', 'stale'].every((arm) => + arms[arm].correctness.rate === null + || arms.empty.correctness.rate === null + || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + ); + const gates = { + minimumPairs: naturalVsEmpty.pairs >= minimumPairs, + capture: captureRate !== null && captureRate >= 0.8, + recurrenceMagnitude: + relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, + recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, + correctness: + (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, + negativeControls: controlsSafe, + }; + const claimReady = Object.values(gates).every(Boolean); + return { + key, + producerClient: rows[0]?.producer?.client || 'unknown', + producerModel: rows[0]?.producer?.model || null, + consumerClient: rows[0]?.consumer?.client || 'unknown', + consumerModel: rows[0]?.consumer?.model || null, + scenarioId: rows[0]?.scenarioId || null, + arms, + effects: { naturalVsEmpty, naturalVsOracle }, + captureRate, + preActionDeliveryRate, + relativeRecurrenceReduction, + gates, + evidenceStatus: claimReady + ? 'pre-registered transfer gates passed' + : `insufficient or failed transfer gates (need ${minimumPairs} pairs)`, + }; + }); +} + +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; + +function concurrencySummary(runs) { + const natural = runs.filter((run) => run.arm === 'natural'); + const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); + const captures = natural.reduce((sum, run) => sum + (run.captureSuccesses || 0), 0); + const integrityPasses = natural.filter((run) => + run.integrity?.zeroLoss + && run.integrity?.parseable + && run.integrity?.orphanedFindings === 0 + ).length; + const delivered = natural.reduce((sum, run) => sum + (run.delivery?.delivered || 0), 0); + const expected = natural.reduce((sum, run) => sum + (run.delivery?.expected || 0), 0); + const effect = handoffPairedEffect(runs, 'empty', 'natural'); + return { + runs: runs.length, + naturalRuns: natural.length, + writers, + captureRate: writers ? captures / writers : null, + integrityPassRate: natural.length ? integrityPasses / natural.length : null, + deliveryCoverage: expected ? delivered / expected : null, + naturalCorrectness: proportionInterval( + natural.filter((run) => run.consumer?.correct).length, natural.length + ), + effect, + }; +} + /** * The dashboard-facing evidence console. Deterministic fixture verification, * live hook traces, and randomized model evals remain visibly distinct: only @@ -1088,6 +1245,8 @@ function matchesFilters(event, filters) { export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { const evidence = readEvidence(dir).filter((event) => matchesFilters(event, filters)); const runs = evidence.filter((event) => event.kind === 'eval-run'); + const handoffRuns = evidence.filter((event) => event.kind === 'handoff-run'); + const concurrencyRuns = evidence.filter((event) => event.kind === 'concurrency-run'); const injections = evidence.filter((event) => event.kind === 'inject'); const outcomes = evidence.filter((event) => event.kind === 'tool-outcome'); const feedback = evidence.filter((event) => event.kind === 'finding-feedback'); @@ -1157,8 +1316,10 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { byInjection.has(injection.injectionId) ).length; const harmful = feedback.filter((event) => event.rating === 'harmful').length; + const transferCohorts = handoffCohorts(handoffRuns); + const concurrency = concurrencySummary(concurrencyRuns); return { - schemaVersion: 2, + schemaVersion: 3, generatedAt: new Date().toISOString(), filters, summary: { @@ -1166,6 +1327,8 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { joinedOutcomes: joined, causalJoinCoverage: injections.length ? joined / injections.length : null, evalRuns: runs.length, + handoffRuns: handoffRuns.length, + concurrencyRuns: concurrencyRuns.length, cohorts: cohorts.length, harmfulFeedback: harmful, harmRate: feedback.length ? harmful / feedback.length : null, @@ -1174,12 +1337,16 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { : 'insufficient randomized evidence', }, cohorts, + transferCohorts, + concurrency, episodes: traced, methodology: { intervals: 'deterministic percentile bootstrap (95%); Wilson interval for correctness', causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, + handoffMinimumPairs: + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/kilo/hooks/lib/adapter.mjs b/integrations/kilo/hooks/lib/adapter.mjs index 2bb0082b9..c99d512cc 100644 --- a/integrations/kilo/hooks/lib/adapter.mjs +++ b/integrations/kilo/hooks/lib/adapter.mjs @@ -52,6 +52,8 @@ import { forSharedCommand, forTouch, noteActClasses, + sessionIndex, + standingRules, } from './inject.mjs'; import { indexFile } from './staleness.mjs'; import { isArchived } from './transcript.mjs'; @@ -420,7 +422,30 @@ export async function run(clientName, event) { if (event === 'session-start') { if (!features.routing && !features.retrieval && !features.harvest) process.exit(0); - const output = contextOutput(client, eventName, policyText(client.canDeny)); + // Some hook hosts invoke SessionStart without closing stdin. Waiting the + // full pre-tool timeout would turn optional context into a five-second + // startup tax; lifecycle payloads are tiny and arrive immediately when + // the host supplies one. + const raw = await readStdin({ timeoutMs: 250 }) || {}; + const parts = [policyText(client.canDeny)]; + if (features.retrieval) { + try { + const cwd = raw.cwd || raw.working_directory || process.cwd(); + const dir = wikiDir(projectRootFor(join(cwd, '__session__'), cwd)); + // SessionStart needs hashes and claims, not stored file bodies. Parsing + // the snapshot sidecar here would make startup scale with repository + // history even though this compact index never renders a diff. + const graph = load(dir); + const episode = episodeMeta({ client: clientName, raw }); + const rules = standingRules(dir, graph); + if (rules) parts.push(rules); + const index = sessionIndex(dir, graph, { episode }); + if (index) parts.push(index); + } catch { + // Retrieval is optional context; the policy must still reach the model. + } + } + const output = contextOutput(client, eventName, parts.join('\n\n')); if (output) emit(output); process.exit(0); } diff --git a/integrations/kilo/hooks/lib/inject.mjs b/integrations/kilo/hooks/lib/inject.mjs index 993752c44..db1ecec4a 100644 --- a/integrations/kilo/hooks/lib/inject.mjs +++ b/integrations/kilo/hooks/lib/inject.mjs @@ -779,13 +779,21 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph) { +export function sessionIndex(dir, graph, { episode = {} } = {}) { const budget = indexBudget(dir); // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. const findings = [...graph.nodes.values()] - .filter((n) => n.kind === 'finding' && !n.retired && typeof n.claim === 'string'); + .filter((n) => + n.kind === 'finding' + && !n.retired + && typeof n.claim === 'string' + // These are already rendered in full by `standingRules`; listing them a + // second time spends tokens without adding information. + && n.pinned !== true + && !(n.type === 'feedback' && n.origin === 'human') + ); if (!findings.length) return null; const now = Date.now(); @@ -793,20 +801,68 @@ export function sessionIndex(dir, graph) { ((b.confidence || 0.5) / (1 + (now - (b.at || now)) / 2.6e9)) - ((a.confidence || 0.5) / (1 + (now - (a.at || now)) / 2.6e9))); - const lines = []; - let spent = 0; + const selected = []; for (const finding of ranked) { - const line = `- ${finding.key}: ${finding.claim.slice(0, 90)}`; // claim guaranteed above - const cost = estimate(line); - if (spent + cost > budget) break; - lines.push(line); - spent += cost; + // Staleness checks can touch disk, so bound the candidates BEFORE serving + // them rather than checking an arbitrarily large graph on SessionStart. + const candidate = [...selected, finding]; + const preview = renderSessionIndex(findings.length, candidate); + if (estimate(preview) > budget) break; + selected.push(finding); } - if (!lines.length) return null; + if (!selected.length) return null; + + // `serve` is the only path allowed to hand a finding to a model. In + // particular, activating this previously-unwired index must not create a new + // path that labels an invalidated content claim as current. + const served = serve(graph, selected); + if (!served.length) return null; + // A stale marker is longer than the fresh preview used for candidate + // selection. Trim from the lowest-ranked end until the ACTUAL message fits. + while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + served.pop(); + } + if (!served.length) return null; + const text = renderSessionIndex(findings.length, served); + const spent = estimate(text); + const findingIds = served.map((finding) => finding.key); + + record(dir, { + ...episode, + kind: 'index', + surface: 'session-start', + count: served.length, + tokens: spent, + findingIds, + staleCount: served.filter((finding) => finding.stale).length, + }); + record(dir, { + ...episode, + kind: 'inject', + surface: 'session-start', + anchor: 'session-index', + holdout: false, + tokens: spent, + deliveredTokens: spent, + shadowTokens: spent, + count: served.length, + candidateCount: selected.length, + findingIds, + shadowFindingIds: findingIds, + stale: served.some((finding) => finding.stale), + }); - record(dir, { kind: 'index', count: lines.length, tokens: spent }); + return text; +} - return `# Project wiki (${findings.length} findings, ${lines.length} listed) +function renderSessionIndex(total, findings) { + const lines = findings.map((finding) => { + const freshness = finding.stale + ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` + : ''; + return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; + }); + return `# Project wiki (${total} findings, ${findings.length} listed) Established in previous sessions. Call wiki_query with a key for detail, or just work -- findings anchored to a file are surfaced automatically when you touch it. diff --git a/integrations/kilo/hooks/lib/metrics.mjs b/integrations/kilo/hooks/lib/metrics.mjs index 4fe1882ed..3e4c4bb59 100644 --- a/integrations/kilo/hooks/lib/metrics.mjs +++ b/integrations/kilo/hooks/lib/metrics.mjs @@ -98,6 +98,8 @@ export const EVIDENCE_KINDS = new Set([ 'tool-outcome', 'episode-outcome', 'eval-run', + 'handoff-run', + 'concurrency-run', 'finding-feedback', 'retrieval-decision', 'mcp-client', @@ -1075,11 +1077,166 @@ function pairedEffects(runs, controlArm, treatmentArm, contrastType = 'increment function matchesFilters(event, filters) { for (const field of ['client', 'clientVersion', 'model', 'modelVersion', 'taskId', 'arm']) { - if (filters[field] && String(event[field] || '') !== String(filters[field])) return false; + if (!filters[field]) continue; + let actual = event[field]; + if (event.kind === 'handoff-run') { + actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + } else if (event.kind === 'concurrency-run') { + actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + } + if (String(actual || '') !== String(filters[field])) return false; } return true; } +function handoffArmMetrics(runs) { + const consumers = runs.map((run) => run.consumer || {}); + const count = (field) => consumers.filter((consumer) => consumer[field] === true).length; + const interval = (field) => bootstrapMeanInterval( + consumers.map((consumer) => consumer[field]).filter((value) => numeric(value) !== null), + { seed: `handoff:${field}` } + ); + return { + runs: runs.length, + correctness: proportionInterval(count('correct'), runs.length), + firstPass: proportionInterval(count('firstPass'), runs.length), + mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), + mistakeExecuted: proportionInterval(count('mistakeExecuted'), runs.length), + totalTokens: interval('totalTokens'), + toolCalls: interval('toolCalls'), + failedToolCalls: interval('failedToolCalls'), + latencyMs: interval('latencyMs'), + }; +} + +function handoffPairedEffect(runs, controlArm, treatmentArm) { + const pairs = new Map(); + for (const run of runs) { + if (!run.pairId || ![controlArm, treatmentArm].includes(run.arm)) continue; + if (!pairs.has(run.pairId)) pairs.set(run.pairId, {}); + pairs.get(run.pairId)[run.arm] = run; + } + const complete = [...pairs.values()].filter((pair) => pair[controlArm] && pair[treatmentArm]); + const effect = (field, positiveWhenLower = true) => bootstrapMeanInterval( + complete.map((pair) => { + const control = numeric(pair[controlArm].consumer?.[field]); + const treatment = numeric(pair[treatmentArm].consumer?.[field]); + if (control === null || treatment === null) return null; + return positiveWhenLower ? control - treatment : treatment - control; + }).filter((value) => value !== null), + { seed: `handoff:${controlArm}:${treatmentArm}:${field}` } + ); + return { + comparison: `${treatmentArm} vs ${controlArm}`, + pairs: complete.length, + attemptedMistakesPrevented: effect('mistakeAttempted'), + executedMistakesPrevented: effect('mistakeExecuted'), + firstPassDelta: effect('firstPass', false), + correctnessDelta: effect('correct', false), + totalTokensSaved: effect('totalTokens'), + toolCallsAvoided: effect('toolCalls'), + failedToolCallsAvoided: effect('failedToolCalls'), + latencyMsSaved: effect('latencyMs'), + }; +} + +const handoffKey = (run) => [ + run.producer?.client || 'unknown', run.producer?.model || 'unknown', + run.consumer?.client || 'unknown', run.consumer?.model || 'unknown', + run.scenarioId || 'unknown', +].join('|'); + +function handoffCohorts(runs) { + const grouped = new Map(); + for (const run of runs) { + const key = handoffKey(run); + if (!grouped.has(key)) grouped.set(key, []); + grouped.get(key).push(run); + } + const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + return [...grouped.entries()].map(([key, rows]) => { + const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ + arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), + ])); + const naturalRows = rows.filter((row) => row.arm === 'natural'); + const naturalVsEmpty = handoffPairedEffect(rows, 'empty', 'natural'); + const naturalVsOracle = handoffPairedEffect(rows, 'oracle', 'natural'); + const captureRate = naturalRows.length + ? naturalRows.filter((row) => row.producer?.captureSuccess).length / naturalRows.length + : null; + const preActionDeliveryRate = naturalRows.length + ? naturalRows.filter((row) => row.delivery?.beforeFirstExecutedMistake).length / naturalRows.length + : null; + const emptyRecurrence = arms.empty.mistakeExecuted.rate; + const naturalRecurrence = arms.natural.mistakeExecuted.rate; + const relativeRecurrenceReduction = emptyRecurrence + ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence + : null; + const controlsSafe = ['irrelevant', 'stale'].every((arm) => + arms[arm].correctness.rate === null + || arms.empty.correctness.rate === null + || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + ); + const gates = { + minimumPairs: naturalVsEmpty.pairs >= minimumPairs, + capture: captureRate !== null && captureRate >= 0.8, + recurrenceMagnitude: + relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, + recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, + correctness: + (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, + negativeControls: controlsSafe, + }; + const claimReady = Object.values(gates).every(Boolean); + return { + key, + producerClient: rows[0]?.producer?.client || 'unknown', + producerModel: rows[0]?.producer?.model || null, + consumerClient: rows[0]?.consumer?.client || 'unknown', + consumerModel: rows[0]?.consumer?.model || null, + scenarioId: rows[0]?.scenarioId || null, + arms, + effects: { naturalVsEmpty, naturalVsOracle }, + captureRate, + preActionDeliveryRate, + relativeRecurrenceReduction, + gates, + evidenceStatus: claimReady + ? 'pre-registered transfer gates passed' + : `insufficient or failed transfer gates (need ${minimumPairs} pairs)`, + }; + }); +} + +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; + +function concurrencySummary(runs) { + const natural = runs.filter((run) => run.arm === 'natural'); + const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); + const captures = natural.reduce((sum, run) => sum + (run.captureSuccesses || 0), 0); + const integrityPasses = natural.filter((run) => + run.integrity?.zeroLoss + && run.integrity?.parseable + && run.integrity?.orphanedFindings === 0 + ).length; + const delivered = natural.reduce((sum, run) => sum + (run.delivery?.delivered || 0), 0); + const expected = natural.reduce((sum, run) => sum + (run.delivery?.expected || 0), 0); + const effect = handoffPairedEffect(runs, 'empty', 'natural'); + return { + runs: runs.length, + naturalRuns: natural.length, + writers, + captureRate: writers ? captures / writers : null, + integrityPassRate: natural.length ? integrityPasses / natural.length : null, + deliveryCoverage: expected ? delivered / expected : null, + naturalCorrectness: proportionInterval( + natural.filter((run) => run.consumer?.correct).length, natural.length + ), + effect, + }; +} + /** * The dashboard-facing evidence console. Deterministic fixture verification, * live hook traces, and randomized model evals remain visibly distinct: only @@ -1088,6 +1245,8 @@ function matchesFilters(event, filters) { export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { const evidence = readEvidence(dir).filter((event) => matchesFilters(event, filters)); const runs = evidence.filter((event) => event.kind === 'eval-run'); + const handoffRuns = evidence.filter((event) => event.kind === 'handoff-run'); + const concurrencyRuns = evidence.filter((event) => event.kind === 'concurrency-run'); const injections = evidence.filter((event) => event.kind === 'inject'); const outcomes = evidence.filter((event) => event.kind === 'tool-outcome'); const feedback = evidence.filter((event) => event.kind === 'finding-feedback'); @@ -1157,8 +1316,10 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { byInjection.has(injection.injectionId) ).length; const harmful = feedback.filter((event) => event.rating === 'harmful').length; + const transferCohorts = handoffCohorts(handoffRuns); + const concurrency = concurrencySummary(concurrencyRuns); return { - schemaVersion: 2, + schemaVersion: 3, generatedAt: new Date().toISOString(), filters, summary: { @@ -1166,6 +1327,8 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { joinedOutcomes: joined, causalJoinCoverage: injections.length ? joined / injections.length : null, evalRuns: runs.length, + handoffRuns: handoffRuns.length, + concurrencyRuns: concurrencyRuns.length, cohorts: cohorts.length, harmfulFeedback: harmful, harmRate: feedback.length ? harmful / feedback.length : null, @@ -1174,12 +1337,16 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { : 'insufficient randomized evidence', }, cohorts, + transferCohorts, + concurrency, episodes: traced, methodology: { intervals: 'deterministic percentile bootstrap (95%); Wilson interval for correctness', causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, + handoffMinimumPairs: + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/opencode/hooks/lib/adapter.mjs b/integrations/opencode/hooks/lib/adapter.mjs index 2bb0082b9..c99d512cc 100644 --- a/integrations/opencode/hooks/lib/adapter.mjs +++ b/integrations/opencode/hooks/lib/adapter.mjs @@ -52,6 +52,8 @@ import { forSharedCommand, forTouch, noteActClasses, + sessionIndex, + standingRules, } from './inject.mjs'; import { indexFile } from './staleness.mjs'; import { isArchived } from './transcript.mjs'; @@ -420,7 +422,30 @@ export async function run(clientName, event) { if (event === 'session-start') { if (!features.routing && !features.retrieval && !features.harvest) process.exit(0); - const output = contextOutput(client, eventName, policyText(client.canDeny)); + // Some hook hosts invoke SessionStart without closing stdin. Waiting the + // full pre-tool timeout would turn optional context into a five-second + // startup tax; lifecycle payloads are tiny and arrive immediately when + // the host supplies one. + const raw = await readStdin({ timeoutMs: 250 }) || {}; + const parts = [policyText(client.canDeny)]; + if (features.retrieval) { + try { + const cwd = raw.cwd || raw.working_directory || process.cwd(); + const dir = wikiDir(projectRootFor(join(cwd, '__session__'), cwd)); + // SessionStart needs hashes and claims, not stored file bodies. Parsing + // the snapshot sidecar here would make startup scale with repository + // history even though this compact index never renders a diff. + const graph = load(dir); + const episode = episodeMeta({ client: clientName, raw }); + const rules = standingRules(dir, graph); + if (rules) parts.push(rules); + const index = sessionIndex(dir, graph, { episode }); + if (index) parts.push(index); + } catch { + // Retrieval is optional context; the policy must still reach the model. + } + } + const output = contextOutput(client, eventName, parts.join('\n\n')); if (output) emit(output); process.exit(0); } diff --git a/integrations/opencode/hooks/lib/inject.mjs b/integrations/opencode/hooks/lib/inject.mjs index 993752c44..db1ecec4a 100644 --- a/integrations/opencode/hooks/lib/inject.mjs +++ b/integrations/opencode/hooks/lib/inject.mjs @@ -779,13 +779,21 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph) { +export function sessionIndex(dir, graph, { episode = {} } = {}) { const budget = indexBudget(dir); // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. const findings = [...graph.nodes.values()] - .filter((n) => n.kind === 'finding' && !n.retired && typeof n.claim === 'string'); + .filter((n) => + n.kind === 'finding' + && !n.retired + && typeof n.claim === 'string' + // These are already rendered in full by `standingRules`; listing them a + // second time spends tokens without adding information. + && n.pinned !== true + && !(n.type === 'feedback' && n.origin === 'human') + ); if (!findings.length) return null; const now = Date.now(); @@ -793,20 +801,68 @@ export function sessionIndex(dir, graph) { ((b.confidence || 0.5) / (1 + (now - (b.at || now)) / 2.6e9)) - ((a.confidence || 0.5) / (1 + (now - (a.at || now)) / 2.6e9))); - const lines = []; - let spent = 0; + const selected = []; for (const finding of ranked) { - const line = `- ${finding.key}: ${finding.claim.slice(0, 90)}`; // claim guaranteed above - const cost = estimate(line); - if (spent + cost > budget) break; - lines.push(line); - spent += cost; + // Staleness checks can touch disk, so bound the candidates BEFORE serving + // them rather than checking an arbitrarily large graph on SessionStart. + const candidate = [...selected, finding]; + const preview = renderSessionIndex(findings.length, candidate); + if (estimate(preview) > budget) break; + selected.push(finding); } - if (!lines.length) return null; + if (!selected.length) return null; + + // `serve` is the only path allowed to hand a finding to a model. In + // particular, activating this previously-unwired index must not create a new + // path that labels an invalidated content claim as current. + const served = serve(graph, selected); + if (!served.length) return null; + // A stale marker is longer than the fresh preview used for candidate + // selection. Trim from the lowest-ranked end until the ACTUAL message fits. + while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + served.pop(); + } + if (!served.length) return null; + const text = renderSessionIndex(findings.length, served); + const spent = estimate(text); + const findingIds = served.map((finding) => finding.key); + + record(dir, { + ...episode, + kind: 'index', + surface: 'session-start', + count: served.length, + tokens: spent, + findingIds, + staleCount: served.filter((finding) => finding.stale).length, + }); + record(dir, { + ...episode, + kind: 'inject', + surface: 'session-start', + anchor: 'session-index', + holdout: false, + tokens: spent, + deliveredTokens: spent, + shadowTokens: spent, + count: served.length, + candidateCount: selected.length, + findingIds, + shadowFindingIds: findingIds, + stale: served.some((finding) => finding.stale), + }); - record(dir, { kind: 'index', count: lines.length, tokens: spent }); + return text; +} - return `# Project wiki (${findings.length} findings, ${lines.length} listed) +function renderSessionIndex(total, findings) { + const lines = findings.map((finding) => { + const freshness = finding.stale + ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` + : ''; + return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; + }); + return `# Project wiki (${total} findings, ${findings.length} listed) Established in previous sessions. Call wiki_query with a key for detail, or just work -- findings anchored to a file are surfaced automatically when you touch it. diff --git a/integrations/opencode/hooks/lib/metrics.mjs b/integrations/opencode/hooks/lib/metrics.mjs index 4fe1882ed..3e4c4bb59 100644 --- a/integrations/opencode/hooks/lib/metrics.mjs +++ b/integrations/opencode/hooks/lib/metrics.mjs @@ -98,6 +98,8 @@ export const EVIDENCE_KINDS = new Set([ 'tool-outcome', 'episode-outcome', 'eval-run', + 'handoff-run', + 'concurrency-run', 'finding-feedback', 'retrieval-decision', 'mcp-client', @@ -1075,11 +1077,166 @@ function pairedEffects(runs, controlArm, treatmentArm, contrastType = 'increment function matchesFilters(event, filters) { for (const field of ['client', 'clientVersion', 'model', 'modelVersion', 'taskId', 'arm']) { - if (filters[field] && String(event[field] || '') !== String(filters[field])) return false; + if (!filters[field]) continue; + let actual = event[field]; + if (event.kind === 'handoff-run') { + actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + } else if (event.kind === 'concurrency-run') { + actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + } + if (String(actual || '') !== String(filters[field])) return false; } return true; } +function handoffArmMetrics(runs) { + const consumers = runs.map((run) => run.consumer || {}); + const count = (field) => consumers.filter((consumer) => consumer[field] === true).length; + const interval = (field) => bootstrapMeanInterval( + consumers.map((consumer) => consumer[field]).filter((value) => numeric(value) !== null), + { seed: `handoff:${field}` } + ); + return { + runs: runs.length, + correctness: proportionInterval(count('correct'), runs.length), + firstPass: proportionInterval(count('firstPass'), runs.length), + mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), + mistakeExecuted: proportionInterval(count('mistakeExecuted'), runs.length), + totalTokens: interval('totalTokens'), + toolCalls: interval('toolCalls'), + failedToolCalls: interval('failedToolCalls'), + latencyMs: interval('latencyMs'), + }; +} + +function handoffPairedEffect(runs, controlArm, treatmentArm) { + const pairs = new Map(); + for (const run of runs) { + if (!run.pairId || ![controlArm, treatmentArm].includes(run.arm)) continue; + if (!pairs.has(run.pairId)) pairs.set(run.pairId, {}); + pairs.get(run.pairId)[run.arm] = run; + } + const complete = [...pairs.values()].filter((pair) => pair[controlArm] && pair[treatmentArm]); + const effect = (field, positiveWhenLower = true) => bootstrapMeanInterval( + complete.map((pair) => { + const control = numeric(pair[controlArm].consumer?.[field]); + const treatment = numeric(pair[treatmentArm].consumer?.[field]); + if (control === null || treatment === null) return null; + return positiveWhenLower ? control - treatment : treatment - control; + }).filter((value) => value !== null), + { seed: `handoff:${controlArm}:${treatmentArm}:${field}` } + ); + return { + comparison: `${treatmentArm} vs ${controlArm}`, + pairs: complete.length, + attemptedMistakesPrevented: effect('mistakeAttempted'), + executedMistakesPrevented: effect('mistakeExecuted'), + firstPassDelta: effect('firstPass', false), + correctnessDelta: effect('correct', false), + totalTokensSaved: effect('totalTokens'), + toolCallsAvoided: effect('toolCalls'), + failedToolCallsAvoided: effect('failedToolCalls'), + latencyMsSaved: effect('latencyMs'), + }; +} + +const handoffKey = (run) => [ + run.producer?.client || 'unknown', run.producer?.model || 'unknown', + run.consumer?.client || 'unknown', run.consumer?.model || 'unknown', + run.scenarioId || 'unknown', +].join('|'); + +function handoffCohorts(runs) { + const grouped = new Map(); + for (const run of runs) { + const key = handoffKey(run); + if (!grouped.has(key)) grouped.set(key, []); + grouped.get(key).push(run); + } + const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + return [...grouped.entries()].map(([key, rows]) => { + const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ + arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), + ])); + const naturalRows = rows.filter((row) => row.arm === 'natural'); + const naturalVsEmpty = handoffPairedEffect(rows, 'empty', 'natural'); + const naturalVsOracle = handoffPairedEffect(rows, 'oracle', 'natural'); + const captureRate = naturalRows.length + ? naturalRows.filter((row) => row.producer?.captureSuccess).length / naturalRows.length + : null; + const preActionDeliveryRate = naturalRows.length + ? naturalRows.filter((row) => row.delivery?.beforeFirstExecutedMistake).length / naturalRows.length + : null; + const emptyRecurrence = arms.empty.mistakeExecuted.rate; + const naturalRecurrence = arms.natural.mistakeExecuted.rate; + const relativeRecurrenceReduction = emptyRecurrence + ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence + : null; + const controlsSafe = ['irrelevant', 'stale'].every((arm) => + arms[arm].correctness.rate === null + || arms.empty.correctness.rate === null + || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + ); + const gates = { + minimumPairs: naturalVsEmpty.pairs >= minimumPairs, + capture: captureRate !== null && captureRate >= 0.8, + recurrenceMagnitude: + relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, + recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, + correctness: + (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, + negativeControls: controlsSafe, + }; + const claimReady = Object.values(gates).every(Boolean); + return { + key, + producerClient: rows[0]?.producer?.client || 'unknown', + producerModel: rows[0]?.producer?.model || null, + consumerClient: rows[0]?.consumer?.client || 'unknown', + consumerModel: rows[0]?.consumer?.model || null, + scenarioId: rows[0]?.scenarioId || null, + arms, + effects: { naturalVsEmpty, naturalVsOracle }, + captureRate, + preActionDeliveryRate, + relativeRecurrenceReduction, + gates, + evidenceStatus: claimReady + ? 'pre-registered transfer gates passed' + : `insufficient or failed transfer gates (need ${minimumPairs} pairs)`, + }; + }); +} + +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; + +function concurrencySummary(runs) { + const natural = runs.filter((run) => run.arm === 'natural'); + const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); + const captures = natural.reduce((sum, run) => sum + (run.captureSuccesses || 0), 0); + const integrityPasses = natural.filter((run) => + run.integrity?.zeroLoss + && run.integrity?.parseable + && run.integrity?.orphanedFindings === 0 + ).length; + const delivered = natural.reduce((sum, run) => sum + (run.delivery?.delivered || 0), 0); + const expected = natural.reduce((sum, run) => sum + (run.delivery?.expected || 0), 0); + const effect = handoffPairedEffect(runs, 'empty', 'natural'); + return { + runs: runs.length, + naturalRuns: natural.length, + writers, + captureRate: writers ? captures / writers : null, + integrityPassRate: natural.length ? integrityPasses / natural.length : null, + deliveryCoverage: expected ? delivered / expected : null, + naturalCorrectness: proportionInterval( + natural.filter((run) => run.consumer?.correct).length, natural.length + ), + effect, + }; +} + /** * The dashboard-facing evidence console. Deterministic fixture verification, * live hook traces, and randomized model evals remain visibly distinct: only @@ -1088,6 +1245,8 @@ function matchesFilters(event, filters) { export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { const evidence = readEvidence(dir).filter((event) => matchesFilters(event, filters)); const runs = evidence.filter((event) => event.kind === 'eval-run'); + const handoffRuns = evidence.filter((event) => event.kind === 'handoff-run'); + const concurrencyRuns = evidence.filter((event) => event.kind === 'concurrency-run'); const injections = evidence.filter((event) => event.kind === 'inject'); const outcomes = evidence.filter((event) => event.kind === 'tool-outcome'); const feedback = evidence.filter((event) => event.kind === 'finding-feedback'); @@ -1157,8 +1316,10 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { byInjection.has(injection.injectionId) ).length; const harmful = feedback.filter((event) => event.rating === 'harmful').length; + const transferCohorts = handoffCohorts(handoffRuns); + const concurrency = concurrencySummary(concurrencyRuns); return { - schemaVersion: 2, + schemaVersion: 3, generatedAt: new Date().toISOString(), filters, summary: { @@ -1166,6 +1327,8 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { joinedOutcomes: joined, causalJoinCoverage: injections.length ? joined / injections.length : null, evalRuns: runs.length, + handoffRuns: handoffRuns.length, + concurrencyRuns: concurrencyRuns.length, cohorts: cohorts.length, harmfulFeedback: harmful, harmRate: feedback.length ? harmful / feedback.length : null, @@ -1174,12 +1337,16 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { : 'insufficient randomized evidence', }, cohorts, + transferCohorts, + concurrency, episodes: traced, methodology: { intervals: 'deterministic percentile bootstrap (95%); Wilson interval for correctness', causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, + handoffMinimumPairs: + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/qwen/hooks/lib/adapter.mjs b/integrations/qwen/hooks/lib/adapter.mjs index 2bb0082b9..c99d512cc 100644 --- a/integrations/qwen/hooks/lib/adapter.mjs +++ b/integrations/qwen/hooks/lib/adapter.mjs @@ -52,6 +52,8 @@ import { forSharedCommand, forTouch, noteActClasses, + sessionIndex, + standingRules, } from './inject.mjs'; import { indexFile } from './staleness.mjs'; import { isArchived } from './transcript.mjs'; @@ -420,7 +422,30 @@ export async function run(clientName, event) { if (event === 'session-start') { if (!features.routing && !features.retrieval && !features.harvest) process.exit(0); - const output = contextOutput(client, eventName, policyText(client.canDeny)); + // Some hook hosts invoke SessionStart without closing stdin. Waiting the + // full pre-tool timeout would turn optional context into a five-second + // startup tax; lifecycle payloads are tiny and arrive immediately when + // the host supplies one. + const raw = await readStdin({ timeoutMs: 250 }) || {}; + const parts = [policyText(client.canDeny)]; + if (features.retrieval) { + try { + const cwd = raw.cwd || raw.working_directory || process.cwd(); + const dir = wikiDir(projectRootFor(join(cwd, '__session__'), cwd)); + // SessionStart needs hashes and claims, not stored file bodies. Parsing + // the snapshot sidecar here would make startup scale with repository + // history even though this compact index never renders a diff. + const graph = load(dir); + const episode = episodeMeta({ client: clientName, raw }); + const rules = standingRules(dir, graph); + if (rules) parts.push(rules); + const index = sessionIndex(dir, graph, { episode }); + if (index) parts.push(index); + } catch { + // Retrieval is optional context; the policy must still reach the model. + } + } + const output = contextOutput(client, eventName, parts.join('\n\n')); if (output) emit(output); process.exit(0); } diff --git a/integrations/qwen/hooks/lib/inject.mjs b/integrations/qwen/hooks/lib/inject.mjs index 993752c44..db1ecec4a 100644 --- a/integrations/qwen/hooks/lib/inject.mjs +++ b/integrations/qwen/hooks/lib/inject.mjs @@ -779,13 +779,21 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph) { +export function sessionIndex(dir, graph, { episode = {} } = {}) { const budget = indexBudget(dir); // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. const findings = [...graph.nodes.values()] - .filter((n) => n.kind === 'finding' && !n.retired && typeof n.claim === 'string'); + .filter((n) => + n.kind === 'finding' + && !n.retired + && typeof n.claim === 'string' + // These are already rendered in full by `standingRules`; listing them a + // second time spends tokens without adding information. + && n.pinned !== true + && !(n.type === 'feedback' && n.origin === 'human') + ); if (!findings.length) return null; const now = Date.now(); @@ -793,20 +801,68 @@ export function sessionIndex(dir, graph) { ((b.confidence || 0.5) / (1 + (now - (b.at || now)) / 2.6e9)) - ((a.confidence || 0.5) / (1 + (now - (a.at || now)) / 2.6e9))); - const lines = []; - let spent = 0; + const selected = []; for (const finding of ranked) { - const line = `- ${finding.key}: ${finding.claim.slice(0, 90)}`; // claim guaranteed above - const cost = estimate(line); - if (spent + cost > budget) break; - lines.push(line); - spent += cost; + // Staleness checks can touch disk, so bound the candidates BEFORE serving + // them rather than checking an arbitrarily large graph on SessionStart. + const candidate = [...selected, finding]; + const preview = renderSessionIndex(findings.length, candidate); + if (estimate(preview) > budget) break; + selected.push(finding); } - if (!lines.length) return null; + if (!selected.length) return null; + + // `serve` is the only path allowed to hand a finding to a model. In + // particular, activating this previously-unwired index must not create a new + // path that labels an invalidated content claim as current. + const served = serve(graph, selected); + if (!served.length) return null; + // A stale marker is longer than the fresh preview used for candidate + // selection. Trim from the lowest-ranked end until the ACTUAL message fits. + while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + served.pop(); + } + if (!served.length) return null; + const text = renderSessionIndex(findings.length, served); + const spent = estimate(text); + const findingIds = served.map((finding) => finding.key); + + record(dir, { + ...episode, + kind: 'index', + surface: 'session-start', + count: served.length, + tokens: spent, + findingIds, + staleCount: served.filter((finding) => finding.stale).length, + }); + record(dir, { + ...episode, + kind: 'inject', + surface: 'session-start', + anchor: 'session-index', + holdout: false, + tokens: spent, + deliveredTokens: spent, + shadowTokens: spent, + count: served.length, + candidateCount: selected.length, + findingIds, + shadowFindingIds: findingIds, + stale: served.some((finding) => finding.stale), + }); - record(dir, { kind: 'index', count: lines.length, tokens: spent }); + return text; +} - return `# Project wiki (${findings.length} findings, ${lines.length} listed) +function renderSessionIndex(total, findings) { + const lines = findings.map((finding) => { + const freshness = finding.stale + ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` + : ''; + return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; + }); + return `# Project wiki (${total} findings, ${findings.length} listed) Established in previous sessions. Call wiki_query with a key for detail, or just work -- findings anchored to a file are surfaced automatically when you touch it. diff --git a/integrations/qwen/hooks/lib/metrics.mjs b/integrations/qwen/hooks/lib/metrics.mjs index 4fe1882ed..3e4c4bb59 100644 --- a/integrations/qwen/hooks/lib/metrics.mjs +++ b/integrations/qwen/hooks/lib/metrics.mjs @@ -98,6 +98,8 @@ export const EVIDENCE_KINDS = new Set([ 'tool-outcome', 'episode-outcome', 'eval-run', + 'handoff-run', + 'concurrency-run', 'finding-feedback', 'retrieval-decision', 'mcp-client', @@ -1075,11 +1077,166 @@ function pairedEffects(runs, controlArm, treatmentArm, contrastType = 'increment function matchesFilters(event, filters) { for (const field of ['client', 'clientVersion', 'model', 'modelVersion', 'taskId', 'arm']) { - if (filters[field] && String(event[field] || '') !== String(filters[field])) return false; + if (!filters[field]) continue; + let actual = event[field]; + if (event.kind === 'handoff-run') { + actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + } else if (event.kind === 'concurrency-run') { + actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + } + if (String(actual || '') !== String(filters[field])) return false; } return true; } +function handoffArmMetrics(runs) { + const consumers = runs.map((run) => run.consumer || {}); + const count = (field) => consumers.filter((consumer) => consumer[field] === true).length; + const interval = (field) => bootstrapMeanInterval( + consumers.map((consumer) => consumer[field]).filter((value) => numeric(value) !== null), + { seed: `handoff:${field}` } + ); + return { + runs: runs.length, + correctness: proportionInterval(count('correct'), runs.length), + firstPass: proportionInterval(count('firstPass'), runs.length), + mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), + mistakeExecuted: proportionInterval(count('mistakeExecuted'), runs.length), + totalTokens: interval('totalTokens'), + toolCalls: interval('toolCalls'), + failedToolCalls: interval('failedToolCalls'), + latencyMs: interval('latencyMs'), + }; +} + +function handoffPairedEffect(runs, controlArm, treatmentArm) { + const pairs = new Map(); + for (const run of runs) { + if (!run.pairId || ![controlArm, treatmentArm].includes(run.arm)) continue; + if (!pairs.has(run.pairId)) pairs.set(run.pairId, {}); + pairs.get(run.pairId)[run.arm] = run; + } + const complete = [...pairs.values()].filter((pair) => pair[controlArm] && pair[treatmentArm]); + const effect = (field, positiveWhenLower = true) => bootstrapMeanInterval( + complete.map((pair) => { + const control = numeric(pair[controlArm].consumer?.[field]); + const treatment = numeric(pair[treatmentArm].consumer?.[field]); + if (control === null || treatment === null) return null; + return positiveWhenLower ? control - treatment : treatment - control; + }).filter((value) => value !== null), + { seed: `handoff:${controlArm}:${treatmentArm}:${field}` } + ); + return { + comparison: `${treatmentArm} vs ${controlArm}`, + pairs: complete.length, + attemptedMistakesPrevented: effect('mistakeAttempted'), + executedMistakesPrevented: effect('mistakeExecuted'), + firstPassDelta: effect('firstPass', false), + correctnessDelta: effect('correct', false), + totalTokensSaved: effect('totalTokens'), + toolCallsAvoided: effect('toolCalls'), + failedToolCallsAvoided: effect('failedToolCalls'), + latencyMsSaved: effect('latencyMs'), + }; +} + +const handoffKey = (run) => [ + run.producer?.client || 'unknown', run.producer?.model || 'unknown', + run.consumer?.client || 'unknown', run.consumer?.model || 'unknown', + run.scenarioId || 'unknown', +].join('|'); + +function handoffCohorts(runs) { + const grouped = new Map(); + for (const run of runs) { + const key = handoffKey(run); + if (!grouped.has(key)) grouped.set(key, []); + grouped.get(key).push(run); + } + const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + return [...grouped.entries()].map(([key, rows]) => { + const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ + arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), + ])); + const naturalRows = rows.filter((row) => row.arm === 'natural'); + const naturalVsEmpty = handoffPairedEffect(rows, 'empty', 'natural'); + const naturalVsOracle = handoffPairedEffect(rows, 'oracle', 'natural'); + const captureRate = naturalRows.length + ? naturalRows.filter((row) => row.producer?.captureSuccess).length / naturalRows.length + : null; + const preActionDeliveryRate = naturalRows.length + ? naturalRows.filter((row) => row.delivery?.beforeFirstExecutedMistake).length / naturalRows.length + : null; + const emptyRecurrence = arms.empty.mistakeExecuted.rate; + const naturalRecurrence = arms.natural.mistakeExecuted.rate; + const relativeRecurrenceReduction = emptyRecurrence + ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence + : null; + const controlsSafe = ['irrelevant', 'stale'].every((arm) => + arms[arm].correctness.rate === null + || arms.empty.correctness.rate === null + || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + ); + const gates = { + minimumPairs: naturalVsEmpty.pairs >= minimumPairs, + capture: captureRate !== null && captureRate >= 0.8, + recurrenceMagnitude: + relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, + recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, + correctness: + (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, + negativeControls: controlsSafe, + }; + const claimReady = Object.values(gates).every(Boolean); + return { + key, + producerClient: rows[0]?.producer?.client || 'unknown', + producerModel: rows[0]?.producer?.model || null, + consumerClient: rows[0]?.consumer?.client || 'unknown', + consumerModel: rows[0]?.consumer?.model || null, + scenarioId: rows[0]?.scenarioId || null, + arms, + effects: { naturalVsEmpty, naturalVsOracle }, + captureRate, + preActionDeliveryRate, + relativeRecurrenceReduction, + gates, + evidenceStatus: claimReady + ? 'pre-registered transfer gates passed' + : `insufficient or failed transfer gates (need ${minimumPairs} pairs)`, + }; + }); +} + +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; + +function concurrencySummary(runs) { + const natural = runs.filter((run) => run.arm === 'natural'); + const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); + const captures = natural.reduce((sum, run) => sum + (run.captureSuccesses || 0), 0); + const integrityPasses = natural.filter((run) => + run.integrity?.zeroLoss + && run.integrity?.parseable + && run.integrity?.orphanedFindings === 0 + ).length; + const delivered = natural.reduce((sum, run) => sum + (run.delivery?.delivered || 0), 0); + const expected = natural.reduce((sum, run) => sum + (run.delivery?.expected || 0), 0); + const effect = handoffPairedEffect(runs, 'empty', 'natural'); + return { + runs: runs.length, + naturalRuns: natural.length, + writers, + captureRate: writers ? captures / writers : null, + integrityPassRate: natural.length ? integrityPasses / natural.length : null, + deliveryCoverage: expected ? delivered / expected : null, + naturalCorrectness: proportionInterval( + natural.filter((run) => run.consumer?.correct).length, natural.length + ), + effect, + }; +} + /** * The dashboard-facing evidence console. Deterministic fixture verification, * live hook traces, and randomized model evals remain visibly distinct: only @@ -1088,6 +1245,8 @@ function matchesFilters(event, filters) { export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { const evidence = readEvidence(dir).filter((event) => matchesFilters(event, filters)); const runs = evidence.filter((event) => event.kind === 'eval-run'); + const handoffRuns = evidence.filter((event) => event.kind === 'handoff-run'); + const concurrencyRuns = evidence.filter((event) => event.kind === 'concurrency-run'); const injections = evidence.filter((event) => event.kind === 'inject'); const outcomes = evidence.filter((event) => event.kind === 'tool-outcome'); const feedback = evidence.filter((event) => event.kind === 'finding-feedback'); @@ -1157,8 +1316,10 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { byInjection.has(injection.injectionId) ).length; const harmful = feedback.filter((event) => event.rating === 'harmful').length; + const transferCohorts = handoffCohorts(handoffRuns); + const concurrency = concurrencySummary(concurrencyRuns); return { - schemaVersion: 2, + schemaVersion: 3, generatedAt: new Date().toISOString(), filters, summary: { @@ -1166,6 +1327,8 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { joinedOutcomes: joined, causalJoinCoverage: injections.length ? joined / injections.length : null, evalRuns: runs.length, + handoffRuns: handoffRuns.length, + concurrencyRuns: concurrencyRuns.length, cohorts: cohorts.length, harmfulFeedback: harmful, harmRate: feedback.length ? harmful / feedback.length : null, @@ -1174,12 +1337,16 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { : 'insufficient randomized evidence', }, cohorts, + transferCohorts, + concurrency, episodes: traced, methodology: { intervals: 'deterministic percentile bootstrap (95%); Wilson interval for correctness', causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, + handoffMinimumPairs: + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/windsurf/hooks/lib/adapter.mjs b/integrations/windsurf/hooks/lib/adapter.mjs index 2bb0082b9..c99d512cc 100644 --- a/integrations/windsurf/hooks/lib/adapter.mjs +++ b/integrations/windsurf/hooks/lib/adapter.mjs @@ -52,6 +52,8 @@ import { forSharedCommand, forTouch, noteActClasses, + sessionIndex, + standingRules, } from './inject.mjs'; import { indexFile } from './staleness.mjs'; import { isArchived } from './transcript.mjs'; @@ -420,7 +422,30 @@ export async function run(clientName, event) { if (event === 'session-start') { if (!features.routing && !features.retrieval && !features.harvest) process.exit(0); - const output = contextOutput(client, eventName, policyText(client.canDeny)); + // Some hook hosts invoke SessionStart without closing stdin. Waiting the + // full pre-tool timeout would turn optional context into a five-second + // startup tax; lifecycle payloads are tiny and arrive immediately when + // the host supplies one. + const raw = await readStdin({ timeoutMs: 250 }) || {}; + const parts = [policyText(client.canDeny)]; + if (features.retrieval) { + try { + const cwd = raw.cwd || raw.working_directory || process.cwd(); + const dir = wikiDir(projectRootFor(join(cwd, '__session__'), cwd)); + // SessionStart needs hashes and claims, not stored file bodies. Parsing + // the snapshot sidecar here would make startup scale with repository + // history even though this compact index never renders a diff. + const graph = load(dir); + const episode = episodeMeta({ client: clientName, raw }); + const rules = standingRules(dir, graph); + if (rules) parts.push(rules); + const index = sessionIndex(dir, graph, { episode }); + if (index) parts.push(index); + } catch { + // Retrieval is optional context; the policy must still reach the model. + } + } + const output = contextOutput(client, eventName, parts.join('\n\n')); if (output) emit(output); process.exit(0); } diff --git a/integrations/windsurf/hooks/lib/inject.mjs b/integrations/windsurf/hooks/lib/inject.mjs index 993752c44..db1ecec4a 100644 --- a/integrations/windsurf/hooks/lib/inject.mjs +++ b/integrations/windsurf/hooks/lib/inject.mjs @@ -779,13 +779,21 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph) { +export function sessionIndex(dir, graph, { episode = {} } = {}) { const budget = indexBudget(dir); // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. const findings = [...graph.nodes.values()] - .filter((n) => n.kind === 'finding' && !n.retired && typeof n.claim === 'string'); + .filter((n) => + n.kind === 'finding' + && !n.retired + && typeof n.claim === 'string' + // These are already rendered in full by `standingRules`; listing them a + // second time spends tokens without adding information. + && n.pinned !== true + && !(n.type === 'feedback' && n.origin === 'human') + ); if (!findings.length) return null; const now = Date.now(); @@ -793,20 +801,68 @@ export function sessionIndex(dir, graph) { ((b.confidence || 0.5) / (1 + (now - (b.at || now)) / 2.6e9)) - ((a.confidence || 0.5) / (1 + (now - (a.at || now)) / 2.6e9))); - const lines = []; - let spent = 0; + const selected = []; for (const finding of ranked) { - const line = `- ${finding.key}: ${finding.claim.slice(0, 90)}`; // claim guaranteed above - const cost = estimate(line); - if (spent + cost > budget) break; - lines.push(line); - spent += cost; + // Staleness checks can touch disk, so bound the candidates BEFORE serving + // them rather than checking an arbitrarily large graph on SessionStart. + const candidate = [...selected, finding]; + const preview = renderSessionIndex(findings.length, candidate); + if (estimate(preview) > budget) break; + selected.push(finding); } - if (!lines.length) return null; + if (!selected.length) return null; + + // `serve` is the only path allowed to hand a finding to a model. In + // particular, activating this previously-unwired index must not create a new + // path that labels an invalidated content claim as current. + const served = serve(graph, selected); + if (!served.length) return null; + // A stale marker is longer than the fresh preview used for candidate + // selection. Trim from the lowest-ranked end until the ACTUAL message fits. + while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + served.pop(); + } + if (!served.length) return null; + const text = renderSessionIndex(findings.length, served); + const spent = estimate(text); + const findingIds = served.map((finding) => finding.key); + + record(dir, { + ...episode, + kind: 'index', + surface: 'session-start', + count: served.length, + tokens: spent, + findingIds, + staleCount: served.filter((finding) => finding.stale).length, + }); + record(dir, { + ...episode, + kind: 'inject', + surface: 'session-start', + anchor: 'session-index', + holdout: false, + tokens: spent, + deliveredTokens: spent, + shadowTokens: spent, + count: served.length, + candidateCount: selected.length, + findingIds, + shadowFindingIds: findingIds, + stale: served.some((finding) => finding.stale), + }); - record(dir, { kind: 'index', count: lines.length, tokens: spent }); + return text; +} - return `# Project wiki (${findings.length} findings, ${lines.length} listed) +function renderSessionIndex(total, findings) { + const lines = findings.map((finding) => { + const freshness = finding.stale + ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` + : ''; + return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; + }); + return `# Project wiki (${total} findings, ${findings.length} listed) Established in previous sessions. Call wiki_query with a key for detail, or just work -- findings anchored to a file are surfaced automatically when you touch it. diff --git a/integrations/windsurf/hooks/lib/metrics.mjs b/integrations/windsurf/hooks/lib/metrics.mjs index 4fe1882ed..3e4c4bb59 100644 --- a/integrations/windsurf/hooks/lib/metrics.mjs +++ b/integrations/windsurf/hooks/lib/metrics.mjs @@ -98,6 +98,8 @@ export const EVIDENCE_KINDS = new Set([ 'tool-outcome', 'episode-outcome', 'eval-run', + 'handoff-run', + 'concurrency-run', 'finding-feedback', 'retrieval-decision', 'mcp-client', @@ -1075,11 +1077,166 @@ function pairedEffects(runs, controlArm, treatmentArm, contrastType = 'increment function matchesFilters(event, filters) { for (const field of ['client', 'clientVersion', 'model', 'modelVersion', 'taskId', 'arm']) { - if (filters[field] && String(event[field] || '') !== String(filters[field])) return false; + if (!filters[field]) continue; + let actual = event[field]; + if (event.kind === 'handoff-run') { + actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + } else if (event.kind === 'concurrency-run') { + actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + } + if (String(actual || '') !== String(filters[field])) return false; } return true; } +function handoffArmMetrics(runs) { + const consumers = runs.map((run) => run.consumer || {}); + const count = (field) => consumers.filter((consumer) => consumer[field] === true).length; + const interval = (field) => bootstrapMeanInterval( + consumers.map((consumer) => consumer[field]).filter((value) => numeric(value) !== null), + { seed: `handoff:${field}` } + ); + return { + runs: runs.length, + correctness: proportionInterval(count('correct'), runs.length), + firstPass: proportionInterval(count('firstPass'), runs.length), + mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), + mistakeExecuted: proportionInterval(count('mistakeExecuted'), runs.length), + totalTokens: interval('totalTokens'), + toolCalls: interval('toolCalls'), + failedToolCalls: interval('failedToolCalls'), + latencyMs: interval('latencyMs'), + }; +} + +function handoffPairedEffect(runs, controlArm, treatmentArm) { + const pairs = new Map(); + for (const run of runs) { + if (!run.pairId || ![controlArm, treatmentArm].includes(run.arm)) continue; + if (!pairs.has(run.pairId)) pairs.set(run.pairId, {}); + pairs.get(run.pairId)[run.arm] = run; + } + const complete = [...pairs.values()].filter((pair) => pair[controlArm] && pair[treatmentArm]); + const effect = (field, positiveWhenLower = true) => bootstrapMeanInterval( + complete.map((pair) => { + const control = numeric(pair[controlArm].consumer?.[field]); + const treatment = numeric(pair[treatmentArm].consumer?.[field]); + if (control === null || treatment === null) return null; + return positiveWhenLower ? control - treatment : treatment - control; + }).filter((value) => value !== null), + { seed: `handoff:${controlArm}:${treatmentArm}:${field}` } + ); + return { + comparison: `${treatmentArm} vs ${controlArm}`, + pairs: complete.length, + attemptedMistakesPrevented: effect('mistakeAttempted'), + executedMistakesPrevented: effect('mistakeExecuted'), + firstPassDelta: effect('firstPass', false), + correctnessDelta: effect('correct', false), + totalTokensSaved: effect('totalTokens'), + toolCallsAvoided: effect('toolCalls'), + failedToolCallsAvoided: effect('failedToolCalls'), + latencyMsSaved: effect('latencyMs'), + }; +} + +const handoffKey = (run) => [ + run.producer?.client || 'unknown', run.producer?.model || 'unknown', + run.consumer?.client || 'unknown', run.consumer?.model || 'unknown', + run.scenarioId || 'unknown', +].join('|'); + +function handoffCohorts(runs) { + const grouped = new Map(); + for (const run of runs) { + const key = handoffKey(run); + if (!grouped.has(key)) grouped.set(key, []); + grouped.get(key).push(run); + } + const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + return [...grouped.entries()].map(([key, rows]) => { + const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ + arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), + ])); + const naturalRows = rows.filter((row) => row.arm === 'natural'); + const naturalVsEmpty = handoffPairedEffect(rows, 'empty', 'natural'); + const naturalVsOracle = handoffPairedEffect(rows, 'oracle', 'natural'); + const captureRate = naturalRows.length + ? naturalRows.filter((row) => row.producer?.captureSuccess).length / naturalRows.length + : null; + const preActionDeliveryRate = naturalRows.length + ? naturalRows.filter((row) => row.delivery?.beforeFirstExecutedMistake).length / naturalRows.length + : null; + const emptyRecurrence = arms.empty.mistakeExecuted.rate; + const naturalRecurrence = arms.natural.mistakeExecuted.rate; + const relativeRecurrenceReduction = emptyRecurrence + ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence + : null; + const controlsSafe = ['irrelevant', 'stale'].every((arm) => + arms[arm].correctness.rate === null + || arms.empty.correctness.rate === null + || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + ); + const gates = { + minimumPairs: naturalVsEmpty.pairs >= minimumPairs, + capture: captureRate !== null && captureRate >= 0.8, + recurrenceMagnitude: + relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, + recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, + correctness: + (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, + negativeControls: controlsSafe, + }; + const claimReady = Object.values(gates).every(Boolean); + return { + key, + producerClient: rows[0]?.producer?.client || 'unknown', + producerModel: rows[0]?.producer?.model || null, + consumerClient: rows[0]?.consumer?.client || 'unknown', + consumerModel: rows[0]?.consumer?.model || null, + scenarioId: rows[0]?.scenarioId || null, + arms, + effects: { naturalVsEmpty, naturalVsOracle }, + captureRate, + preActionDeliveryRate, + relativeRecurrenceReduction, + gates, + evidenceStatus: claimReady + ? 'pre-registered transfer gates passed' + : `insufficient or failed transfer gates (need ${minimumPairs} pairs)`, + }; + }); +} + +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; + +function concurrencySummary(runs) { + const natural = runs.filter((run) => run.arm === 'natural'); + const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); + const captures = natural.reduce((sum, run) => sum + (run.captureSuccesses || 0), 0); + const integrityPasses = natural.filter((run) => + run.integrity?.zeroLoss + && run.integrity?.parseable + && run.integrity?.orphanedFindings === 0 + ).length; + const delivered = natural.reduce((sum, run) => sum + (run.delivery?.delivered || 0), 0); + const expected = natural.reduce((sum, run) => sum + (run.delivery?.expected || 0), 0); + const effect = handoffPairedEffect(runs, 'empty', 'natural'); + return { + runs: runs.length, + naturalRuns: natural.length, + writers, + captureRate: writers ? captures / writers : null, + integrityPassRate: natural.length ? integrityPasses / natural.length : null, + deliveryCoverage: expected ? delivered / expected : null, + naturalCorrectness: proportionInterval( + natural.filter((run) => run.consumer?.correct).length, natural.length + ), + effect, + }; +} + /** * The dashboard-facing evidence console. Deterministic fixture verification, * live hook traces, and randomized model evals remain visibly distinct: only @@ -1088,6 +1245,8 @@ function matchesFilters(event, filters) { export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { const evidence = readEvidence(dir).filter((event) => matchesFilters(event, filters)); const runs = evidence.filter((event) => event.kind === 'eval-run'); + const handoffRuns = evidence.filter((event) => event.kind === 'handoff-run'); + const concurrencyRuns = evidence.filter((event) => event.kind === 'concurrency-run'); const injections = evidence.filter((event) => event.kind === 'inject'); const outcomes = evidence.filter((event) => event.kind === 'tool-outcome'); const feedback = evidence.filter((event) => event.kind === 'finding-feedback'); @@ -1157,8 +1316,10 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { byInjection.has(injection.injectionId) ).length; const harmful = feedback.filter((event) => event.rating === 'harmful').length; + const transferCohorts = handoffCohorts(handoffRuns); + const concurrency = concurrencySummary(concurrencyRuns); return { - schemaVersion: 2, + schemaVersion: 3, generatedAt: new Date().toISOString(), filters, summary: { @@ -1166,6 +1327,8 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { joinedOutcomes: joined, causalJoinCoverage: injections.length ? joined / injections.length : null, evalRuns: runs.length, + handoffRuns: handoffRuns.length, + concurrencyRuns: concurrencyRuns.length, cohorts: cohorts.length, harmfulFeedback: harmful, harmRate: feedback.length ? harmful / feedback.length : null, @@ -1174,12 +1337,16 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { : 'insufficient randomized evidence', }, cohorts, + transferCohorts, + concurrency, episodes: traced, methodology: { intervals: 'deterministic percentile bootstrap (95%); Wilson interval for correctness', causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, + handoffMinimumPairs: + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), deterministicChecksAreCausalProof: false, }, }; diff --git a/package.json b/package.json index 67824aa4a..0e1421ae8 100644 --- a/package.json +++ b/package.json @@ -62,6 +62,9 @@ "verify:certification": "node scripts/certify-clients.mjs --no-version-check", "eval:evidence": "node scripts/run-evidence-eval.mjs", "eval:evidence:plan": "node scripts/run-evidence-eval.mjs --dry-run", + "eval:handoff": "node scripts/run-handoff-eval.mjs", + "eval:handoff:plan": "node scripts/run-handoff-eval.mjs --dry-run", + "eval:handoff:concurrent": "node scripts/run-concurrent-handoff-eval.mjs", "verify:all": "npm run sync:hooks:check && npm run verify:clients && npm run verify:certification && npm run verify:harvest && npm run verify:live-graph && npm run verify:ui && npm run verify:interactions", "verify:interactions": "node scripts/verify-wiki-interactions.mjs", "explore:wiki": "node scripts/explore-wiki.mjs" diff --git a/plugin/hooks/lib/adapter.mjs b/plugin/hooks/lib/adapter.mjs index 2bb0082b9..c99d512cc 100644 --- a/plugin/hooks/lib/adapter.mjs +++ b/plugin/hooks/lib/adapter.mjs @@ -52,6 +52,8 @@ import { forSharedCommand, forTouch, noteActClasses, + sessionIndex, + standingRules, } from './inject.mjs'; import { indexFile } from './staleness.mjs'; import { isArchived } from './transcript.mjs'; @@ -420,7 +422,30 @@ export async function run(clientName, event) { if (event === 'session-start') { if (!features.routing && !features.retrieval && !features.harvest) process.exit(0); - const output = contextOutput(client, eventName, policyText(client.canDeny)); + // Some hook hosts invoke SessionStart without closing stdin. Waiting the + // full pre-tool timeout would turn optional context into a five-second + // startup tax; lifecycle payloads are tiny and arrive immediately when + // the host supplies one. + const raw = await readStdin({ timeoutMs: 250 }) || {}; + const parts = [policyText(client.canDeny)]; + if (features.retrieval) { + try { + const cwd = raw.cwd || raw.working_directory || process.cwd(); + const dir = wikiDir(projectRootFor(join(cwd, '__session__'), cwd)); + // SessionStart needs hashes and claims, not stored file bodies. Parsing + // the snapshot sidecar here would make startup scale with repository + // history even though this compact index never renders a diff. + const graph = load(dir); + const episode = episodeMeta({ client: clientName, raw }); + const rules = standingRules(dir, graph); + if (rules) parts.push(rules); + const index = sessionIndex(dir, graph, { episode }); + if (index) parts.push(index); + } catch { + // Retrieval is optional context; the policy must still reach the model. + } + } + const output = contextOutput(client, eventName, parts.join('\n\n')); if (output) emit(output); process.exit(0); } diff --git a/plugin/hooks/lib/inject.mjs b/plugin/hooks/lib/inject.mjs index 993752c44..db1ecec4a 100644 --- a/plugin/hooks/lib/inject.mjs +++ b/plugin/hooks/lib/inject.mjs @@ -779,13 +779,21 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph) { +export function sessionIndex(dir, graph, { episode = {} } = {}) { const budget = indexBudget(dir); // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. const findings = [...graph.nodes.values()] - .filter((n) => n.kind === 'finding' && !n.retired && typeof n.claim === 'string'); + .filter((n) => + n.kind === 'finding' + && !n.retired + && typeof n.claim === 'string' + // These are already rendered in full by `standingRules`; listing them a + // second time spends tokens without adding information. + && n.pinned !== true + && !(n.type === 'feedback' && n.origin === 'human') + ); if (!findings.length) return null; const now = Date.now(); @@ -793,20 +801,68 @@ export function sessionIndex(dir, graph) { ((b.confidence || 0.5) / (1 + (now - (b.at || now)) / 2.6e9)) - ((a.confidence || 0.5) / (1 + (now - (a.at || now)) / 2.6e9))); - const lines = []; - let spent = 0; + const selected = []; for (const finding of ranked) { - const line = `- ${finding.key}: ${finding.claim.slice(0, 90)}`; // claim guaranteed above - const cost = estimate(line); - if (spent + cost > budget) break; - lines.push(line); - spent += cost; + // Staleness checks can touch disk, so bound the candidates BEFORE serving + // them rather than checking an arbitrarily large graph on SessionStart. + const candidate = [...selected, finding]; + const preview = renderSessionIndex(findings.length, candidate); + if (estimate(preview) > budget) break; + selected.push(finding); } - if (!lines.length) return null; + if (!selected.length) return null; + + // `serve` is the only path allowed to hand a finding to a model. In + // particular, activating this previously-unwired index must not create a new + // path that labels an invalidated content claim as current. + const served = serve(graph, selected); + if (!served.length) return null; + // A stale marker is longer than the fresh preview used for candidate + // selection. Trim from the lowest-ranked end until the ACTUAL message fits. + while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + served.pop(); + } + if (!served.length) return null; + const text = renderSessionIndex(findings.length, served); + const spent = estimate(text); + const findingIds = served.map((finding) => finding.key); + + record(dir, { + ...episode, + kind: 'index', + surface: 'session-start', + count: served.length, + tokens: spent, + findingIds, + staleCount: served.filter((finding) => finding.stale).length, + }); + record(dir, { + ...episode, + kind: 'inject', + surface: 'session-start', + anchor: 'session-index', + holdout: false, + tokens: spent, + deliveredTokens: spent, + shadowTokens: spent, + count: served.length, + candidateCount: selected.length, + findingIds, + shadowFindingIds: findingIds, + stale: served.some((finding) => finding.stale), + }); - record(dir, { kind: 'index', count: lines.length, tokens: spent }); + return text; +} - return `# Project wiki (${findings.length} findings, ${lines.length} listed) +function renderSessionIndex(total, findings) { + const lines = findings.map((finding) => { + const freshness = finding.stale + ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` + : ''; + return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; + }); + return `# Project wiki (${total} findings, ${findings.length} listed) Established in previous sessions. Call wiki_query with a key for detail, or just work -- findings anchored to a file are surfaced automatically when you touch it. diff --git a/plugin/hooks/lib/metrics.mjs b/plugin/hooks/lib/metrics.mjs index 4fe1882ed..3e4c4bb59 100644 --- a/plugin/hooks/lib/metrics.mjs +++ b/plugin/hooks/lib/metrics.mjs @@ -98,6 +98,8 @@ export const EVIDENCE_KINDS = new Set([ 'tool-outcome', 'episode-outcome', 'eval-run', + 'handoff-run', + 'concurrency-run', 'finding-feedback', 'retrieval-decision', 'mcp-client', @@ -1075,11 +1077,166 @@ function pairedEffects(runs, controlArm, treatmentArm, contrastType = 'increment function matchesFilters(event, filters) { for (const field of ['client', 'clientVersion', 'model', 'modelVersion', 'taskId', 'arm']) { - if (filters[field] && String(event[field] || '') !== String(filters[field])) return false; + if (!filters[field]) continue; + let actual = event[field]; + if (event.kind === 'handoff-run') { + actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + } else if (event.kind === 'concurrency-run') { + actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + } + if (String(actual || '') !== String(filters[field])) return false; } return true; } +function handoffArmMetrics(runs) { + const consumers = runs.map((run) => run.consumer || {}); + const count = (field) => consumers.filter((consumer) => consumer[field] === true).length; + const interval = (field) => bootstrapMeanInterval( + consumers.map((consumer) => consumer[field]).filter((value) => numeric(value) !== null), + { seed: `handoff:${field}` } + ); + return { + runs: runs.length, + correctness: proportionInterval(count('correct'), runs.length), + firstPass: proportionInterval(count('firstPass'), runs.length), + mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), + mistakeExecuted: proportionInterval(count('mistakeExecuted'), runs.length), + totalTokens: interval('totalTokens'), + toolCalls: interval('toolCalls'), + failedToolCalls: interval('failedToolCalls'), + latencyMs: interval('latencyMs'), + }; +} + +function handoffPairedEffect(runs, controlArm, treatmentArm) { + const pairs = new Map(); + for (const run of runs) { + if (!run.pairId || ![controlArm, treatmentArm].includes(run.arm)) continue; + if (!pairs.has(run.pairId)) pairs.set(run.pairId, {}); + pairs.get(run.pairId)[run.arm] = run; + } + const complete = [...pairs.values()].filter((pair) => pair[controlArm] && pair[treatmentArm]); + const effect = (field, positiveWhenLower = true) => bootstrapMeanInterval( + complete.map((pair) => { + const control = numeric(pair[controlArm].consumer?.[field]); + const treatment = numeric(pair[treatmentArm].consumer?.[field]); + if (control === null || treatment === null) return null; + return positiveWhenLower ? control - treatment : treatment - control; + }).filter((value) => value !== null), + { seed: `handoff:${controlArm}:${treatmentArm}:${field}` } + ); + return { + comparison: `${treatmentArm} vs ${controlArm}`, + pairs: complete.length, + attemptedMistakesPrevented: effect('mistakeAttempted'), + executedMistakesPrevented: effect('mistakeExecuted'), + firstPassDelta: effect('firstPass', false), + correctnessDelta: effect('correct', false), + totalTokensSaved: effect('totalTokens'), + toolCallsAvoided: effect('toolCalls'), + failedToolCallsAvoided: effect('failedToolCalls'), + latencyMsSaved: effect('latencyMs'), + }; +} + +const handoffKey = (run) => [ + run.producer?.client || 'unknown', run.producer?.model || 'unknown', + run.consumer?.client || 'unknown', run.consumer?.model || 'unknown', + run.scenarioId || 'unknown', +].join('|'); + +function handoffCohorts(runs) { + const grouped = new Map(); + for (const run of runs) { + const key = handoffKey(run); + if (!grouped.has(key)) grouped.set(key, []); + grouped.get(key).push(run); + } + const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + return [...grouped.entries()].map(([key, rows]) => { + const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ + arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), + ])); + const naturalRows = rows.filter((row) => row.arm === 'natural'); + const naturalVsEmpty = handoffPairedEffect(rows, 'empty', 'natural'); + const naturalVsOracle = handoffPairedEffect(rows, 'oracle', 'natural'); + const captureRate = naturalRows.length + ? naturalRows.filter((row) => row.producer?.captureSuccess).length / naturalRows.length + : null; + const preActionDeliveryRate = naturalRows.length + ? naturalRows.filter((row) => row.delivery?.beforeFirstExecutedMistake).length / naturalRows.length + : null; + const emptyRecurrence = arms.empty.mistakeExecuted.rate; + const naturalRecurrence = arms.natural.mistakeExecuted.rate; + const relativeRecurrenceReduction = emptyRecurrence + ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence + : null; + const controlsSafe = ['irrelevant', 'stale'].every((arm) => + arms[arm].correctness.rate === null + || arms.empty.correctness.rate === null + || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + ); + const gates = { + minimumPairs: naturalVsEmpty.pairs >= minimumPairs, + capture: captureRate !== null && captureRate >= 0.8, + recurrenceMagnitude: + relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, + recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, + correctness: + (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, + negativeControls: controlsSafe, + }; + const claimReady = Object.values(gates).every(Boolean); + return { + key, + producerClient: rows[0]?.producer?.client || 'unknown', + producerModel: rows[0]?.producer?.model || null, + consumerClient: rows[0]?.consumer?.client || 'unknown', + consumerModel: rows[0]?.consumer?.model || null, + scenarioId: rows[0]?.scenarioId || null, + arms, + effects: { naturalVsEmpty, naturalVsOracle }, + captureRate, + preActionDeliveryRate, + relativeRecurrenceReduction, + gates, + evidenceStatus: claimReady + ? 'pre-registered transfer gates passed' + : `insufficient or failed transfer gates (need ${minimumPairs} pairs)`, + }; + }); +} + +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; + +function concurrencySummary(runs) { + const natural = runs.filter((run) => run.arm === 'natural'); + const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); + const captures = natural.reduce((sum, run) => sum + (run.captureSuccesses || 0), 0); + const integrityPasses = natural.filter((run) => + run.integrity?.zeroLoss + && run.integrity?.parseable + && run.integrity?.orphanedFindings === 0 + ).length; + const delivered = natural.reduce((sum, run) => sum + (run.delivery?.delivered || 0), 0); + const expected = natural.reduce((sum, run) => sum + (run.delivery?.expected || 0), 0); + const effect = handoffPairedEffect(runs, 'empty', 'natural'); + return { + runs: runs.length, + naturalRuns: natural.length, + writers, + captureRate: writers ? captures / writers : null, + integrityPassRate: natural.length ? integrityPasses / natural.length : null, + deliveryCoverage: expected ? delivered / expected : null, + naturalCorrectness: proportionInterval( + natural.filter((run) => run.consumer?.correct).length, natural.length + ), + effect, + }; +} + /** * The dashboard-facing evidence console. Deterministic fixture verification, * live hook traces, and randomized model evals remain visibly distinct: only @@ -1088,6 +1245,8 @@ function matchesFilters(event, filters) { export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { const evidence = readEvidence(dir).filter((event) => matchesFilters(event, filters)); const runs = evidence.filter((event) => event.kind === 'eval-run'); + const handoffRuns = evidence.filter((event) => event.kind === 'handoff-run'); + const concurrencyRuns = evidence.filter((event) => event.kind === 'concurrency-run'); const injections = evidence.filter((event) => event.kind === 'inject'); const outcomes = evidence.filter((event) => event.kind === 'tool-outcome'); const feedback = evidence.filter((event) => event.kind === 'finding-feedback'); @@ -1157,8 +1316,10 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { byInjection.has(injection.injectionId) ).length; const harmful = feedback.filter((event) => event.rating === 'harmful').length; + const transferCohorts = handoffCohorts(handoffRuns); + const concurrency = concurrencySummary(concurrencyRuns); return { - schemaVersion: 2, + schemaVersion: 3, generatedAt: new Date().toISOString(), filters, summary: { @@ -1166,6 +1327,8 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { joinedOutcomes: joined, causalJoinCoverage: injections.length ? joined / injections.length : null, evalRuns: runs.length, + handoffRuns: handoffRuns.length, + concurrencyRuns: concurrencyRuns.length, cohorts: cohorts.length, harmfulFeedback: harmful, harmRate: feedback.length ? harmful / feedback.length : null, @@ -1174,12 +1337,16 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { : 'insufficient randomized evidence', }, cohorts, + transferCohorts, + concurrency, episodes: traced, methodology: { intervals: 'deterministic percentile bootstrap (95%); Wilson interval for correctness', causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, + handoffMinimumPairs: + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), deterministicChecksAreCausalProof: false, }, }; diff --git a/plugin/hooks/session-start.mjs b/plugin/hooks/session-start.mjs index 8b25e1a4e..eb51b3950 100644 --- a/plugin/hooks/session-start.mjs +++ b/plugin/hooks/session-start.mjs @@ -21,8 +21,9 @@ import { mode, MODE_OFF, readPayload, loadState } from './lib/policy.mjs'; import { policyText } from './lib/adapter.mjs'; import { restorationPlan } from './lib/restore.mjs'; -import { standingRules } from './lib/inject.mjs'; +import { sessionIndex, standingRules } from './lib/inject.mjs'; import { wikiDir, load, projectRootFor } from './lib/wiki.mjs'; +import { episodeMeta, featuresForArm } from './lib/experiment.mjs'; import { join } from 'node:path'; if (mode() === MODE_OFF) process.exit(0); @@ -40,12 +41,11 @@ if (mode() === MODE_OFF) process.exit(0); * is the one place restoration text cannot survive. The `compact` source is the * harness telling us exactly this case. */ -async function restoration() { +async function restoration(parsed) { // `readPayload` returns the PARSED object, not the raw text. Parsing it again // throws on `[object Object]`, and the throw is swallowed by the catch below, // so the restoration block simply never appeared -- the failure mode a // fail-open hook is most likely to hide. - const parsed = await readPayload(); if (!parsed) return null; // Only after a compaction. A cold start has nothing to restore, and paying @@ -81,6 +81,8 @@ async function restoration() { // clients and silently skipped Claude Code, the one client most users are on. // Claude Code keeps its own entry point because its PreToolUse router does more // than the shared one; the standing notice is not a place it needs to differ. +const payload = await readPayload() || {}; +const features = featuresForArm(); const parts = [policyText(true)]; // THE ALWAYS-ON HALF OF DELIVERY. Trigger-fired injection answers "this @@ -93,10 +95,19 @@ const parts = [policyText(true)]; // corrections qualify. An always-on block that grows with the project is how it // becomes wallpaper, and a model stops reading what it always sees. try { - const cwd = process.env.CLAUDE_PROJECT_DIR || process.cwd(); + const cwd = payload.cwd || process.env.CLAUDE_PROJECT_DIR || process.cwd(); const dir = wikiDir(projectRootFor(join(cwd, '__session__'), cwd)); - const rules = standingRules(dir, load(dir)); - if (rules) parts.push(rules); + if (features.retrieval) { + // The index renders claims and freshness state, never snapshot bodies. + // Skipping the sidecar keeps startup bounded on mature graphs. + const graph = load(dir); + const rules = standingRules(dir, graph); + if (rules) parts.push(rules); + const index = sessionIndex(dir, graph, { + episode: episodeMeta({ client: 'claude-code', raw: payload }), + }); + if (index) parts.push(index); + } } catch { // The policy notice must still arrive if the graph is unreadable. } @@ -105,7 +116,7 @@ try { // speaks only to a session resuming from a compaction, so it reads as the more // specific note after the general one. try { - const restored = await restoration(); + const restored = await restoration(payload); if (restored) parts.push(restored); } catch { // The policy notice must still arrive if anything above fails. diff --git a/scripts/run-concurrent-handoff-eval.mjs b/scripts/run-concurrent-handoff-eval.mjs new file mode 100644 index 000000000..7c5d264fc --- /dev/null +++ b/scripts/run-concurrent-handoff-eval.mjs @@ -0,0 +1,287 @@ +#!/usr/bin/env node +/** Live concurrency proof: several producer agents, one graph, one later client. */ + +import { + appendFileSync, cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, + rmSync, +} from 'node:fs'; +import { createHash } from 'node:crypto'; +import { dirname, isAbsolute, join, resolve } from 'node:path'; +import { tmpdir } from 'node:os'; +import { fileURLToPath } from 'node:url'; +import { + extractToolTrace, findNaturalCapture, gradeBehavior, readJsonl, runClient, +} from './run-handoff-eval.mjs'; +import { parseRunIdentity, parseUsage } from './run-evidence-eval.mjs'; +import { audit as auditGraph } from '../hooks-core/curate.mjs'; +import { record } from '../hooks-core/metrics.mjs'; +import { load as loadGraph, wikiDir } from '../hooks-core/wiki.mjs'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +const hash = (value) => createHash('sha256').update(String(value)).digest('hex'); +const fromRoot = (path) => isAbsolute(path) ? path : resolve(ROOT, path); +const readJson = (path) => JSON.parse(readFileSync(path, 'utf8')); + +function options(argv) { + const out = { + suite: join(ROOT, 'evals', 'handoff-suite.json'), + repetitions: 1, + output: join(ROOT, '.token-optimizer', 'evals', `concurrent-handoff-${Date.now()}.jsonl`), + graphDir: wikiDir(ROOT), + keepWorkspaces: false, + }; + for (let index = 0; index < argv.length; index++) { + const arg = argv[index]; + if (arg === '--keep-workspaces') out.keepWorkspaces = true; + else if (arg.startsWith('--')) { + const key = arg.slice(2).replace(/-([a-z])/g, (_, letter) => letter.toUpperCase()); + const value = argv[++index]; + if (value === undefined) throw new Error(`${arg} requires a value`); + out[key] = value; + } + } + out.repetitions = Math.max(1, Number(out.repetitions) || 1); + return out; +} + +function parseIntegrityFile(path) { + if (!existsSync(path)) return { lines: 0, malformed: 0 }; + let lines = 0; + let malformed = 0; + for (const line of readFileSync(path, 'utf8').split('\n')) { + if (!line) continue; + lines += 1; + try { JSON.parse(line); } catch { malformed += 1; } + } + return { lines, malformed }; +} + +export function graphIntegrity(graphDir, expectedFindingIds = []) { + const graph = loadGraph(graphDir); + const audit = auditGraph(graph); + const log = parseIntegrityFile(join(graphDir, 'graph.jsonl')); + const snapshots = parseIntegrityFile(join(graphDir, 'snapshots.jsonl')); + const activeKeys = new Set( + [...graph.nodes.values()].filter((node) => node.kind === 'finding' && !node.retired) + .map((node) => node.key) + ); + const missingAcceptedWrites = expectedFindingIds.filter((key) => !activeKeys.has(key)); + const acceptanceObservable = expectedFindingIds.length > 0; + return { + graphLines: log.lines, + snapshotLines: snapshots.lines, + malformedLines: log.malformed + snapshots.malformed, + orphanedFindings: audit.orphaned.length, + contradictedFindings: audit.contradicted.length, + activeFindings: activeKeys.size, + acceptedWritesObserved: expectedFindingIds.length, + acceptanceObservable, + missingAcceptedWrites, + zeroLoss: acceptanceObservable && missingAcceptedWrites.length === 0, + parseable: log.malformed + snapshots.malformed === 0, + }; +} + +function combinedPrompt() { + return [ + 'Continue all three independent beta follow-ups in this fixture.', + '1. Set src/beta.json state to ready and verify beta with the supported project command.', + '2. Change the beta policy from legacy to modern through its maintainable source of truth, regenerate outputs, and verify synchronization.', + '3. Set targets/beta.json ready to true and validate it with evidence that beta was actually exercised.', + 'Avoid repeating dead ends already established by earlier agents. Complete and verify every item.', + ].join(' '); +} + +function validateProfile(profile, name) { + if (!profile?.command || !Array.isArray(profile.args) || !profile.treatmentConfiguration) { + throw new Error(`runner profile ${name} must declare command, args, and treatmentConfiguration`); + } +} + +async function main() { + const cli = options(process.argv.slice(2)); + if (!cli.runner || !cli.producer || !cli.consumer) { + throw new Error('--runner, --producer, and --consumer are required'); + } + const suite = readJson(fromRoot(cli.suite)); + const scenarios = suite.scenarios || []; + if (scenarios.length < 3) throw new Error('concurrency suite requires at least three scenarios'); + const runners = readJson(fromRoot(cli.runner)); + const producerProfile = runners[cli.producer]; + const consumerProfile = runners[cli.consumer]; + validateProfile(producerProfile, cli.producer); + validateProfile(consumerProfile, cli.consumer); + const aggregate = fromRoot(cli.graphDir); + const output = fromRoot(cli.output); + mkdirSync(aggregate, { recursive: true }); + mkdirSync(dirname(output), { recursive: true }); + + for (let repetition = 0; repetition < cli.repetitions; repetition++) { + const pairId = `concurrent-${repetition + 1}`; + const root = mkdtempSync(join(tmpdir(), `token-optimizer-${pairId}-`)); + const workspace = join(root, 'workspace'); + cpSync(fromRoot(scenarios[0].fixture), workspace, { recursive: true }); + const graphDir = join(root, 'shared-graph'); + const auditPath = join(root, 'producer-audit.jsonl'); + mkdirSync(graphDir, { recursive: true }); + + // Promise.all is deliberate: these are separate CLI processes appending to + // the same graph and metrics logs, not sequential calls labeled concurrent. + const producers = await Promise.all(scenarios.map((scenario) => runClient({ + profile: producerProfile, + client: cli.producer, + model: cli.producerModel || producerProfile.model, + prompt: scenario.producerPrompt, + phase: `writer-${scenario.id}`, + scenario, + pairId, + workspace, + graphDir, + auditPath, + }))); + const producerAudit = readJsonl(auditPath); + const graph = loadGraph(graphDir); + const captures = scenarios.map((scenario) => + findNaturalCapture(graph, scenario.targetFinding) + ); + const findingIds = [...new Set(captures.flat().map((finding) => finding.key))]; + const acceptedFindingIds = [...new Set( + producers.flatMap((producer) => producer.acceptedFindingIds) + )]; + const producerRows = producers.map((producer, index) => { + const scenario = scenarios[index]; + const identity = parseRunIdentity(producer.run.stdout); + const usage = parseUsage(`${producer.run.stdout}\n${producer.run.stderr}`, producerProfile); + return { + scenarioId: scenario.id, + episodeId: producer.episodeId, + client: cli.producer, + clientVersion: identity.clientVersion || producerProfile.version || null, + model: cli.producerModel || producerProfile.model || null, + modelVersion: identity.modelVersion || producerProfile.modelVersion || null, + ...gradeBehavior(scenario, 'producer', workspace, producerAudit, producer.trace), + captureSuccess: captures[index].length > 0, + capturedFindingIds: captures[index].map((finding) => finding.key), + acceptedFindingIds: producer.acceptedFindingIds, + exitCode: producer.run.exitCode, + timedOut: producer.run.timedOut, + latencyMs: producer.run.latencyMs, + ...usage, + toolCalls: usage.toolCalls ?? producer.trace.length, + failedToolCalls: usage.failedToolCalls + ?? producer.trace.filter((action) => action.success === false).length, + }; + }); + const integrity = graphIntegrity(graphDir, acceptedFindingIds); + const frozen = join(root, 'frozen'); + cpSync(workspace, frozen, { recursive: true }); + const arms = repetition % 2 ? ['natural', 'empty'] : ['empty', 'natural']; + + for (const [order, arm] of arms.entries()) { + const consumerWorkspace = join(root, `consumer-${arm}`); + cpSync(frozen, consumerWorkspace, { recursive: true }); + const consumerGraph = join(root, `consumer-graph-${arm}`); + mkdirSync(consumerGraph, { recursive: true }); + if (arm === 'natural') cpSync(graphDir, consumerGraph, { recursive: true }); + const consumerAuditPath = join(root, `consumer-audit-${arm}.jsonl`); + const synthetic = { id: 'concurrent-combined', family: 'concurrent-multi-writer' }; + const consumer = await runClient({ + profile: consumerProfile, + client: cli.consumer, + model: cli.consumerModel || consumerProfile.model, + prompt: combinedPrompt(), + phase: `concurrent-consumer-${arm}`, + scenario: synthetic, + pairId, + workspace: consumerWorkspace, + graphDir: consumerGraph, + auditPath: consumerAuditPath, + }); + const consumerAudit = readJsonl(consumerAuditPath); + const behaviors = scenarios.map((scenario) => + gradeBehavior(scenario, 'consumer', consumerWorkspace, consumerAudit, consumer.trace) + ); + const deliveryEvents = consumer.evidence.filter((event) => + event.kind === 'inject' && event.surface === 'session-start' + ); + const deliveredIds = new Set(deliveryEvents.flatMap((event) => event.findingIds || [])); + const identity = parseRunIdentity(consumer.run.stdout); + const usage = parseUsage( + `${consumer.run.stdout}\n${consumer.run.stderr}`, consumerProfile + ); + const result = { + kind: 'concurrency-run', + schemaVersion: 3, + suite: suite.name, + pairId, + repetition, + order, + arm, + writerCount: producers.length, + producerClient: cli.producer, + producerModel: cli.producerModel || producerProfile.model || null, + writers: producerRows, + captureSuccesses: producerRows.filter((row) => row.captureSuccess).length, + acceptedFindingIds, + integrity, + consumer: { + client: cli.consumer, + clientVersion: identity.clientVersion || consumerProfile.version || null, + model: cli.consumerModel || consumerProfile.model || null, + modelVersion: identity.modelVersion || consumerProfile.modelVersion || null, + sessionId: identity.sessionId, + correct: behaviors.every((behavior) => behavior.correct), + firstPass: behaviors.every((behavior) => behavior.firstPass), + mistakeAttempted: behaviors.some((behavior) => behavior.mistakeAttempted), + mistakeExecuted: behaviors.some((behavior) => behavior.mistakeExecuted), + scenarioResults: Object.fromEntries( + scenarios.map((scenario, index) => [scenario.id, behaviors[index]]) + ), + exitCode: consumer.run.exitCode, + timedOut: consumer.run.timedOut, + latencyMs: consumer.run.latencyMs, + ...usage, + toolCalls: usage.toolCalls ?? consumer.trace.length, + failedToolCalls: usage.failedToolCalls + ?? consumer.trace.filter((action) => action.success === false).length, + }, + delivery: { + expected: arm === 'natural' ? findingIds.length : 0, + delivered: arm === 'natural' + ? findingIds.filter((id) => deliveredIds.has(id)).length + : 0, + allBeforeFirstAction: arm === 'natural' + ? findingIds.every((id) => deliveredIds.has(id)) + : null, + deliveredTokens: deliveryEvents.reduce( + (sum, event) => sum + (event.deliveredTokens || 0), 0 + ), + }, + graphHash: hash( + existsSync(join(graphDir, 'graph.jsonl')) + ? readFileSync(join(graphDir, 'graph.jsonl'), 'utf8') + : 'empty' + ), + producerPromptHashes: scenarios.map((scenario) => hash(scenario.producerPrompt)), + consumerPromptHash: hash(combinedPrompt()), + }; + appendFileSync(output, `${JSON.stringify(result)}\n`); + record(aggregate, result); + process.stdout.write( + `${result.consumer.correct ? 'PASS' : 'FAIL'} ${pairId} ${cli.producer}x${producers.length}` + + `->${cli.consumer} ${arm} captures=${result.captureSuccesses}/${producers.length}` + + ` delivered=${result.delivery.delivered}/${result.delivery.expected}` + + ` malformed=${integrity.malformedLines} lost=${integrity.missingAcceptedWrites.length}\n` + ); + } + + if (!cli.keepWorkspaces) rmSync(root, { recursive: true, force: true }); + } + process.stdout.write(`Concurrency evidence artifact: ${output}\n`); +} + +const isMain = process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url); +if (isMain) main().catch((error) => { + console.error(error instanceof Error ? error.stack || error.message : String(error)); + process.exit(1); +}); diff --git a/scripts/run-evidence-eval.mjs b/scripts/run-evidence-eval.mjs index eeee7b6fe..bcb6830c7 100644 --- a/scripts/run-evidence-eval.mjs +++ b/scripts/run-evidence-eval.mjs @@ -211,19 +211,30 @@ export function parseUsage(text, profile = {}) { } export function parseRunIdentity(text) { + let identity = { modelVersion: null, clientVersion: null, sessionId: null }; for (const line of String(text).split('\n')) { try { const document = JSON.parse(line); if (document?.type === 'system' && document?.subtype === 'init') { - return { + identity = { modelVersion: document.model || null, clientVersion: document.claude_code_version || null, sessionId: document.session_id || null, }; } + // Codex JSONL identifies the ephemeral thread separately from turn + // events. Keep scanning because future versions may add explicit model + // or client metadata later in the stream. + if (document?.type === 'thread.started') { + identity.sessionId = document.thread_id || document.threadId || identity.sessionId; + } + identity.modelVersion = document.model_version || document.modelVersion + || document.model || identity.modelVersion; + identity.clientVersion = document.client_version || document.clientVersion + || identity.clientVersion; } catch { /* non-JSON output */ } } - return { modelVersion: null, clientVersion: null, sessionId: null }; + return identity; } async function grade(task, run, workspace, graphDir, evidenceEvents) { diff --git a/scripts/run-handoff-eval.mjs b/scripts/run-handoff-eval.mjs new file mode 100644 index 000000000..9c341a661 --- /dev/null +++ b/scripts/run-handoff-eval.mjs @@ -0,0 +1,605 @@ +#!/usr/bin/env node +/** + * Paired live-model evaluation for cross-client mistake transfer. + * + * One producer encounters a required dead end and is responsible for semantic + * harvesting. Every consumer arm then receives the SAME frozen post-producer + * workspace. The only intended difference is its graph: empty, natural, + * oracle, irrelevant, or explicitly stale. Grading uses filesystem state, + * fixture audit sentinels, and tool traces; model prose is never the oracle. + */ + +import { + appendFileSync, cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, + readdirSync, rmSync, +} from 'node:fs'; +import { spawn, spawnSync } from 'node:child_process'; +import { createHash } from 'node:crypto'; +import { dirname, isAbsolute, join, relative, resolve } from 'node:path'; +import { tmpdir } from 'node:os'; +import { fileURLToPath } from 'node:url'; +import { parseRunIdentity, parseUsage } from './run-evidence-eval.mjs'; +import { record, readEvidence } from '../hooks-core/metrics.mjs'; +import { load as loadGraph, putNode, wikiDir } from '../hooks-core/wiki.mjs'; +import { writeHarvested } from '../hooks-core/harvest-write.mjs'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +export const HANDOFF_ARMS = Object.freeze(['empty', 'natural', 'oracle', 'irrelevant', 'stale']); + +const git = spawnSync('git', ['rev-parse', 'HEAD'], { + cwd: ROOT, encoding: 'utf8', windowsHide: true, +}); +const status = spawnSync('git', ['status', '--porcelain'], { + cwd: ROOT, encoding: 'utf8', windowsHide: true, +}); +const PROVENANCE = { + commit: git.status === 0 ? git.stdout.trim() : null, + dirty: status.status === 0 ? Boolean(status.stdout.trim()) : null, +}; + +const hash = (value) => createHash('sha256').update(String(value)).digest('hex'); +const json = (path) => JSON.parse(readFileSync(path, 'utf8')); +const fromRoot = (path) => isAbsolute(path) ? path : resolve(ROOT, path); + +function parseArgs(argv) { + const options = { + suite: join(ROOT, 'evals', 'handoff-suite.json'), + repetitions: 1, + output: join(ROOT, '.token-optimizer', 'evals', `handoff-${Date.now()}.jsonl`), + graphDir: wikiDir(ROOT), + dryRun: false, + keepWorkspaces: false, + includeTranscript: false, + }; + for (let index = 0; index < argv.length; index++) { + const arg = argv[index]; + if (arg === '--dry-run') options.dryRun = true; + else if (arg === '--keep-workspaces') options.keepWorkspaces = true; + else if (arg === '--include-transcript') options.includeTranscript = true; + else if (arg.startsWith('--')) { + const key = arg.slice(2).replace(/-([a-z])/g, (_, letter) => letter.toUpperCase()); + const value = argv[++index]; + if (value === undefined) throw new Error(`${arg} requires a value`); + options[key] = value; + } + } + options.repetitions = Math.max(1, Number(options.repetitions) || 1); + return options; +} + +/** Generalized cyclic Latin rotation for any declared arm count. */ +export function handoffSchedule(scenarios, repetitions, arms = HANDOFF_ARMS) { + const schedule = []; + for (const scenario of scenarios) { + for (let repetition = 0; repetition < repetitions; repetition++) { + const order = arms.map((_, index) => arms[(index + repetition) % arms.length]); + schedule.push({ + scenario, + repetition, + pairId: `${scenario.id}-${repetition + 1}`, + arms: order, + }); + } + } + return schedule; +} + +function replace(value, context) { + return String(value).replace( + /\{(prompt|workspace|phase|client|model|scenarioId|pairId)\}/g, + (_, key) => String(context[key] ?? '') + ); +} + +function execute(command, args, { cwd, env, timeoutMs }) { + return new Promise((finish) => { + const started = Date.now(); + const child = spawn(command, args, { + cwd, env, shell: false, windowsHide: true, stdio: ['ignore', 'pipe', 'pipe'], + }); + const stdout = []; + const stderr = []; + let timedOut = false; + const timer = setTimeout(() => { + timedOut = true; + child.kill(); + }, timeoutMs); + child.stdout.on('data', (chunk) => stdout.push(chunk)); + child.stderr.on('data', (chunk) => stderr.push(chunk)); + child.on('error', (error) => { + clearTimeout(timer); + finish({ + exitCode: null, signal: null, timedOut, latencyMs: Date.now() - started, + stdout: Buffer.concat(stdout).toString('utf8'), + stderr: `${Buffer.concat(stderr).toString('utf8')}\n${error.message}`.trim(), + }); + }); + child.on('close', (exitCode, signal) => { + clearTimeout(timer); + finish({ + exitCode, signal, timedOut, latencyMs: Date.now() - started, + stdout: Buffer.concat(stdout).toString('utf8'), + stderr: Buffer.concat(stderr).toString('utf8'), + }); + }); + }); +} + +export function readJsonl(path) { + if (!existsSync(path)) return []; + return readFileSync(path, 'utf8').split('\n').filter(Boolean).flatMap((line) => { + try { return [JSON.parse(line)]; } catch { return []; } + }); +} + +function toolObjects(value, out = []) { + if (!value || typeof value !== 'object') return out; + const type = String(value.type || value.kind || ''); + if ( + type === 'tool_use' + || type === 'command_execution' + || type === 'file_change' + || type === 'mcp_tool_call' + || type === 'tool-outcome' + || type === 'mcp-tool' + ) out.push(value); + for (const child of Object.values(value)) toolObjects(child, out); + return out; +} + +/** Redacted tool trace: only action class, command, path, outcome, and order. */ +export function extractToolTrace(text, evidence = []) { + const objects = []; + for (const line of String(text).split('\n')) { + try { toolObjects(JSON.parse(line), objects); } catch { /* mixed CLI output */ } + } + for (const event of evidence) toolObjects(event, objects); + return objects.map((value, order) => { + const input = value.input || value.tool_input || value.arguments || value.item || {}; + const command = input.command || value.command || input.cmd || null; + const paths = [ + input.file_path, input.path, value.anchor, value.path, + ...(Array.isArray(value.changes) ? value.changes.map((change) => change.path) : []), + ].filter((item) => typeof item === 'string'); + return { + order, + tool: value.name || value.toolName || value.tool_name || value.type || value.kind || null, + command: command == null ? null : String(command), + paths, + success: value.success ?? (value.status === 'failed' ? false : null), + at: Number(value.at) || null, + }; + }); +} + +/** Keys acknowledged by successful wiki_write tool results in a CLI stream. */ +export function extractAcceptedFindingIds(text) { + const accepted = new Set(); + const visit = (value) => { + if (typeof value === 'string') { + const trimmed = value.trim(); + if (trimmed.startsWith('{') && trimmed.length < 100_000) { + try { visit(JSON.parse(trimmed)); } catch { /* ordinary tool text */ } + } + return; + } + if (!value || typeof value !== 'object') return; + if ( + value.success === true + && Number(value.written) > 0 + && Array.isArray(value.keys) + ) { + for (const key of value.keys) if (typeof key === 'string') accepted.add(key); + } + for (const child of Object.values(value)) visit(child); + }; + for (const line of String(text).split('\n')) { + try { visit(JSON.parse(line)); } catch { /* mixed CLI output */ } + } + return [...accepted]; +} + +const includes = (value, fragment) => + String(value || '').toLowerCase().includes(String(fragment).toLowerCase()); + +export function targetFindingMatches(finding, target) { + if (!finding || finding.kind !== 'finding' || finding.retired) return false; + const searchable = [ + finding.claim, finding.evidence, finding.applicability, + ...(finding.invalidators || []), finding.trigger, + ].join(' ').toLowerCase(); + const first = (target.claimAny || []).some((term) => searchable.includes(String(term).toLowerCase())); + const second = (target.claimAnySecondary || []).some( + (term) => searchable.includes(String(term).toLowerCase()) + ); + return first && second; +} + +function isTargetFindingContract(node) { + return node.origin === 'agent' + && typeof node.evidence === 'string' && Boolean(node.evidence.trim()) + && typeof node.applicability === 'string' && Boolean(node.applicability.trim()) + && ['verified', 'probable', 'speculative'].includes(node.confidenceLabel) + && Array.isArray(node.invalidators) && node.invalidators.length > 0 + && typeof node.scope === 'string' && Boolean(node.scope); +} + +export function findNaturalCapture(graph, target) { + return [...graph.nodes.values()].filter( + (node) => targetFindingMatches(node, target) && isTargetFindingContract(node) + ); +} + +function traceContains(trace, pattern) { + return trace.some((action) => + pattern.test(String(action.command || '')) + || action.paths.some((path) => pattern.test(String(path))) + ); +} + +function readFixtureJson(workspace, path) { + try { return JSON.parse(readFileSync(join(workspace, path), 'utf8')); } catch { return null; } +} + +/** Hidden behavioral grader; it never scores claims made in the final answer. */ +export function gradeBehavior(scenario, phase, workspace, audit, trace) { + const target = phase === 'producer' + ? scenario.grader.producerTarget + : scenario.grader.consumerTarget; + let correct = false; + let mistakeAttempted = false; + let mistakeExecuted = false; + let proof = false; + + if (scenario.grader.kind === 'verification') { + const bad = new RegExp(`verify-${target}\\.mjs`, 'i'); + mistakeAttempted = traceContains(trace, bad); + mistakeExecuted = audit.some( + (event) => event.kind === 'unsupported-verification' && event.target === target + ); + proof = audit.some( + (event) => event.kind === 'supported-verification' && event.target === target && event.passed + ); + correct = readFixtureJson(workspace, `src/${target}.json`)?.state === 'ready' && proof; + } else if (scenario.grader.kind === 'generated') { + const generated = new RegExp(`clients[\\\\/]${target}[\\\\/]policy\\.txt`, 'i'); + mistakeAttempted = traceContains(trace, generated); + mistakeExecuted = mistakeAttempted; + const synchronized = audit.some( + (event) => event.kind === 'sync-check' && event.synchronized === true + ); + const source = readFileSync(join(workspace, 'source', `${target}-policy.txt`), 'utf8').trim(); + const client = readFileSync(join(workspace, 'clients', target, 'policy.txt'), 'utf8').trim(); + proof = synchronized; + correct = source === 'modern' && client === 'modern' && synchronized; + } else if (scenario.grader.kind === 'validation') { + const bad = new RegExp(`check-${target}\\.mjs`, 'i'); + mistakeAttempted = traceContains(trace, bad); + mistakeExecuted = audit.some( + (event) => event.kind === 'false-positive-validation' && event.target === target + ); + proof = audit.some( + (event) => event.kind === 'scoped-validation' + && event.target === target && event.exercised && event.passed + ); + correct = readFixtureJson(workspace, `targets/${target}.json`)?.ready === true && proof; + } + + return { + correct, + firstPass: correct && !mistakeExecuted, + mistakeAttempted, + mistakeExecuted, + proof, + }; +} + +function materialize(fixture, prefix) { + const workspace = mkdtempSync(join(tmpdir(), prefix)); + cpSync(fromRoot(fixture), workspace, { recursive: true }); + return workspace; +} + +function frozenSnapshot(producerWorkspace, pairRoot) { + const frozen = join(pairRoot, 'frozen-workspace'); + cpSync(producerWorkspace, frozen, { recursive: true }); + for (const name of ['.git', '.claude', '.codex', '.token-optimizer']) { + rmSync(join(frozen, name), { recursive: true, force: true }); + } + return frozen; +} + +function copyDir(source, destination) { + mkdirSync(destination, { recursive: true }); + if (existsSync(source)) cpSync(source, destination, { recursive: true }); +} + +function seedArm(arm, scenario, workspace, graphDir) { + if (arm === 'empty' || arm === 'natural') return []; + const source = arm === 'irrelevant' ? scenario.irrelevantFinding : scenario.oracleFinding; + const finding = { + ...source, + ...(arm === 'stale' ? { type: 'finding' } : {}), + anchors: source.anchors.map((anchor) => join(workspace, anchor)), + }; + const keys = writeHarvested(graphDir, [finding], { + sessionId: `${arm}-control`, + projectRoot: workspace, + }); + if (arm === 'stale') { + const graph = loadGraph(graphDir); + const node = [...graph.nodes.values()].find((candidate) => candidate.key === keys[0]); + if (node) { + putNode(graphDir, { + ...node, + kind: 'finding', + key: node.key, + stale: true, + staleReason: 'pre-registered stale control', + diff: '- prior control evidence\n+ changed control evidence', + }); + } + } + return keys; +} + +function graphDigest(dir) { + if (!existsSync(dir)) return hash('empty'); + const files = []; + const visit = (current) => { + for (const entry of readdirSync(current, { withFileTypes: true })) { + const path = join(current, entry.name); + if (entry.isDirectory()) visit(path); + else files.push(`${relative(dir, path)}\0${readFileSync(path)}`); + } + }; + visit(dir); + return hash(files.sort().join('\0')); +} + +export async function runClient({ + profile, client, model, prompt, phase, scenario, pairId, workspace, graphDir, auditPath, +}) { + const context = { + prompt, workspace, phase, client, model, scenarioId: scenario.id, pairId, + }; + const args = [...(profile.fullArgs || []), ...(profile.args || [])] + .map((arg) => replace(arg, context)); + const episodeId = `${pairId}-${phase}-${client}`; + const env = { + ...process.env, + ...(profile.env || {}), + TOKEN_OPTIMIZER_EXPERIMENT_ARM: 'full', + TOKEN_OPTIMIZER_EPISODE_ID: episodeId, + TOKEN_OPTIMIZER_PAIR_ID: pairId, + TOKEN_OPTIMIZER_TASK_ID: scenario.id, + TOKEN_OPTIMIZER_MODEL: model || profile.model || '', + TOKEN_OPTIMIZER_CLIENT_VERSION: profile.version || '', + TOKEN_OPTIMIZER_WIKI_DIR: graphDir, + TOKEN_OPTIMIZER_SHARED_DIR: graphDir, + TOKEN_OPTIMIZER_STATE_DIR: join(dirname(graphDir), 'state'), + TOKEN_OPTIMIZER_EVAL_AUDIT: auditPath, + }; + const run = await execute(profile.command, args, { + cwd: workspace, + env, + timeoutMs: Number(profile.timeoutMs) || 600_000, + }); + const evidence = readEvidence(graphDir).filter((event) => event.episodeId === episodeId); + const trace = extractToolTrace(`${run.stdout}\n${run.stderr}`, evidence); + const acceptedFindingIds = extractAcceptedFindingIds(`${run.stdout}\n${run.stderr}`); + return { run, evidence, trace, acceptedFindingIds, episodeId }; +} + +function deliveryGrade(evidence, findingIds, audit, behavior) { + const relevant = evidence.filter((event) => + event.kind === 'inject' + && (event.findingIds || []).some((id) => findingIds.includes(id)) + ); + const firstMistakeAt = audit.find((event) => { + if (behavior.mistakeExecuted && ['unsupported-verification', 'false-positive-validation'].includes(event.kind)) return true; + return behavior.mistakeExecuted && event.kind === 'sync-check' && event.synchronized === false; + })?.at ?? null; + const first = relevant.sort((a, b) => (a.at || 0) - (b.at || 0))[0] || null; + return { + delivered: Boolean(first), + surface: first?.surface || null, + injectionId: first?.injectionId || null, + deliveredTokens: relevant.reduce((sum, event) => sum + (event.deliveredTokens || 0), 0), + beforeFirstExecutedMistake: first + ? (firstMistakeAt === null || Number(first.at) <= Number(firstMistakeAt)) + : false, + }; +} + +function safeEvidence(event) { + const { anchor, cwd, projectRoot, ...safe } = event; + return { + ...safe, + anchorHash: anchor ? hash(anchor) : undefined, + cwdHash: cwd ? hash(cwd) : undefined, + projectRootHash: projectRoot ? hash(projectRoot) : undefined, + }; +} + +function validateProfile(profile, name) { + if (!profile || !profile.command || !Array.isArray(profile.args)) { + throw new Error(`runner profile ${name} must declare command and args`); + } + if (!profile.treatmentConfiguration) { + throw new Error(`runner profile ${name} must document treatmentConfiguration`); + } +} + +async function main() { + const options = parseArgs(process.argv.slice(2)); + const suite = json(fromRoot(options.suite)); + let scenarios = suite.scenarios || []; + if (options.scenarios) { + const selected = new Set(String(options.scenarios).split(',').map((item) => item.trim())); + scenarios = scenarios.filter((scenario) => selected.has(scenario.id)); + } + if (!scenarios.length) throw new Error('no handoff scenarios selected'); + const schedule = handoffSchedule(scenarios, options.repetitions, suite.arms || HANDOFF_ARMS); + if (options.dryRun) { + process.stdout.write(JSON.stringify({ + suite: suite.name, + runs: schedule.map(({ scenario, ...item }) => ({ scenarioId: scenario.id, ...item })), + }, null, 2)); + return; + } + if (!options.runner || !options.producer || !options.consumer) { + throw new Error('--runner, --producer, and --consumer are required unless --dry-run is used'); + } + const runners = json(fromRoot(options.runner)); + const producerProfile = runners[options.producer]; + const consumerProfile = runners[options.consumer]; + validateProfile(producerProfile, options.producer); + validateProfile(consumerProfile, options.consumer); + const aggregateDir = fromRoot(options.graphDir); + const output = fromRoot(options.output); + mkdirSync(aggregateDir, { recursive: true }); + mkdirSync(dirname(output), { recursive: true }); + + for (const item of schedule) { + const pairRoot = mkdtempSync(join(tmpdir(), `token-optimizer-handoff-${item.pairId}-`)); + const producerWorkspace = materialize(item.scenario.fixture, `producer-${item.pairId}-`); + const producerGraph = join(pairRoot, 'producer-graph'); + const producerAuditPath = join(pairRoot, 'producer-audit.jsonl'); + mkdirSync(producerGraph, { recursive: true }); + const producer = await runClient({ + profile: producerProfile, + client: options.producer, + model: options.producerModel || producerProfile.model, + prompt: item.scenario.producerPrompt, + phase: 'producer', + scenario: item.scenario, + pairId: item.pairId, + workspace: producerWorkspace, + graphDir: producerGraph, + auditPath: producerAuditPath, + }); + const producerAudit = readJsonl(producerAuditPath); + const producerBehavior = gradeBehavior( + item.scenario, 'producer', producerWorkspace, producerAudit, producer.trace + ); + const producerGraphState = loadGraph(producerGraph); + const naturalFindings = findNaturalCapture(producerGraphState, item.scenario.targetFinding); + const naturalFindingIds = naturalFindings.map((finding) => finding.key); + const frozen = frozenSnapshot(producerWorkspace, pairRoot); + + for (const [order, arm] of item.arms.entries()) { + const workspace = join(pairRoot, `consumer-${order}-${arm}`); + cpSync(frozen, workspace, { recursive: true }); + const graphDir = join(pairRoot, `graph-${order}-${arm}`); + if (arm === 'natural') copyDir(producerGraph, graphDir); + else mkdirSync(graphDir, { recursive: true }); + const seededFindingIds = seedArm(arm, item.scenario, workspace, graphDir); + const auditPath = join(pairRoot, `audit-${order}-${arm}.jsonl`); + const consumer = await runClient({ + profile: consumerProfile, + client: options.consumer, + model: options.consumerModel || consumerProfile.model, + prompt: item.scenario.consumerPrompt, + phase: `consumer-${arm}`, + scenario: item.scenario, + pairId: item.pairId, + workspace, + graphDir, + auditPath, + }); + const audit = readJsonl(auditPath); + const behavior = gradeBehavior(item.scenario, 'consumer', workspace, audit, consumer.trace); + const expectedFindingIds = arm === 'natural' ? naturalFindingIds : seededFindingIds; + const delivery = deliveryGrade(consumer.evidence, expectedFindingIds, audit, behavior); + const observed = parseRunIdentity(consumer.run.stdout); + const usage = parseUsage(`${consumer.run.stdout}\n${consumer.run.stderr}`, consumerProfile); + const producerIdentity = parseRunIdentity(producer.run.stdout); + const producerUsage = parseUsage( + `${producer.run.stdout}\n${producer.run.stderr}`, producerProfile + ); + const result = { + kind: 'handoff-run', + schemaVersion: 3, + suite: suite.name, + suiteVersion: suite.version, + scenarioId: item.scenario.id, + scenarioFamily: item.scenario.family, + pairId: item.pairId, + repetition: item.repetition, + order, + arm, + producer: { + client: options.producer, + clientVersion: producerIdentity.clientVersion || producerProfile.version || null, + model: options.producerModel || producerProfile.model || null, + modelVersion: producerIdentity.modelVersion || producerProfile.modelVersion || null, + sessionId: producerIdentity.sessionId, + correct: producerBehavior.correct, + mistakeAttempted: producerBehavior.mistakeAttempted, + mistakeExecuted: producerBehavior.mistakeExecuted, + captureSuccess: naturalFindings.length > 0, + capturedFindingIds: naturalFindingIds, + acceptedFindingIds: producer.acceptedFindingIds, + latencyMs: producer.run.latencyMs, + exitCode: producer.run.exitCode, + timedOut: producer.run.timedOut, + ...producerUsage, + toolCalls: producerUsage.toolCalls ?? producer.trace.length, + failedToolCalls: producerUsage.failedToolCalls + ?? producer.trace.filter((action) => action.success === false).length, + }, + consumer: { + client: options.consumer, + clientVersion: observed.clientVersion || consumerProfile.version || null, + model: options.consumerModel || consumerProfile.model || null, + modelVersion: observed.modelVersion || consumerProfile.modelVersion || null, + sessionId: observed.sessionId, + ...behavior, + latencyMs: consumer.run.latencyMs, + exitCode: consumer.run.exitCode, + timedOut: consumer.run.timedOut, + ...usage, + toolCalls: usage.toolCalls ?? consumer.trace.length, + failedToolCalls: usage.failedToolCalls + ?? consumer.trace.filter((action) => action.success === false).length, + }, + delivery, + expectedFindingIds, + naturalCaptureMissing: arm === 'natural' && naturalFindings.length === 0, + graphHash: graphDigest(graphDir), + repoCommit: process.env.GITHUB_SHA || PROVENANCE.commit, + workingTreeDirty: PROVENANCE.dirty, + producerPromptHash: hash(item.scenario.producerPrompt), + consumerPromptHash: hash(item.scenario.consumerPrompt), + fixtureHash: hash(JSON.stringify(item.scenario.grader) + item.scenario.fixture), + stdoutHash: hash(consumer.run.stdout), + stderrHash: hash(consumer.run.stderr), + ...(options.includeTranscript + ? { stdout: consumer.run.stdout, stderr: consumer.run.stderr } + : {}), + }; + appendFileSync(output, `${JSON.stringify(result)}\n`); + record(aggregateDir, result); + for (const event of consumer.evidence) record(aggregateDir, { + ...safeEvidence(event), + evidenceSource: 'live-handoff-trace', + }); + process.stdout.write( + `${behavior.correct ? 'PASS' : 'FAIL'} ${item.scenario.id} ` + + `${options.producer}->${options.consumer} ${arm} ` + + `(mistake=${behavior.mistakeExecuted ? 'yes' : 'no'}, capture=${naturalFindings.length ? 'yes' : 'no'})\n` + ); + } + + rmSync(producerWorkspace, { recursive: true, force: true }); + if (!options.keepWorkspaces) rmSync(pairRoot, { recursive: true, force: true }); + } + process.stdout.write(`Handoff evidence artifact: ${output}\n`); +} + +const isMain = process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url); +if (isMain) main().catch((error) => { + console.error(error instanceof Error ? error.stack || error.message : String(error)); + process.exit(1); +}); diff --git a/scripts/verify-wiki-ui.mjs b/scripts/verify-wiki-ui.mjs index ebea9209c..d04bb5afe 100644 --- a/scripts/verify-wiki-ui.mjs +++ b/scripts/verify-wiki-ui.mjs @@ -101,6 +101,35 @@ async function seed() { }); } } + for (let pair = 1; pair <= 10; pair++) { + for (const arm of ['empty', 'natural', 'oracle', 'irrelevant', 'stale']) { + const prevented = ['natural', 'oracle'].includes(arm); + record(GRAPH, { + kind: 'handoff-run', pairId: `ui-handoff-${pair}`, + scenarioId: 'verification-entry-point', arm, + producer: { client: 'codex', model: 'gpt-5.6-sol', captureSuccess: true }, + consumer: { + client: 'claude-code', model: 'claude-sonnet-5', correct: true, + firstPass: prevented, mistakeAttempted: !prevented, + mistakeExecuted: !prevented, totalTokens: prevented ? 800 : 1000, + toolCalls: prevented ? 5 : 7, failedToolCalls: prevented ? 0 : 1, + }, + delivery: { beforeFirstExecutedMistake: arm === 'natural' }, + }); + } + } + for (const arm of ['empty', 'natural']) { + record(GRAPH, { + kind: 'concurrency-run', pairId: 'ui-concurrent-1', arm, + writerCount: 3, captureSuccesses: 3, + integrity: { zeroLoss: true, parseable: true, orphanedFindings: 0 }, + delivery: { expected: arm === 'natural' ? 3 : 0, delivered: arm === 'natural' ? 3 : 0 }, + consumer: { + client: 'claude-code', model: 'claude-sonnet-5', correct: true, + firstPass: arm === 'natural', mistakeExecuted: arm === 'empty', + }, + }); + } } async function waitForServer(attempts = 40) { @@ -308,6 +337,13 @@ async function main() { check('evidence console distinguishes sufficient causal data', evidenceState === 'ok', evidenceState); const cohortText = await page.textContent('#evidence-cohorts'); check('evidence renders matched effect intervals', /full/i.test(cohortText || '') && /400/.test(cohortText || '')); + const transferText = await page.textContent('#evidence-transfer'); + check('evidence renders cross-client natural transfer separately', + /codex/i.test(transferText || '') && /claude-code/i.test(transferText || '') + && /gates passed/i.test(transferText || '')); + const concurrencyText = await page.textContent('#evidence-concurrency'); + check('evidence renders concurrent writer integrity and later delivery', + /100%/.test(concurrencyText || '') && /3/.test(concurrencyText || '')); const capabilityCount = await page.locator('#evidence-capabilities tbody tr').count(); check('evidence lists all supported client capabilities', capabilityCount === 16, String(capabilityCount)); const capabilityText = await page.textContent('#evidence-capabilities'); diff --git a/src/dashboard/public/js/wiki.js b/src/dashboard/public/js/wiki.js index e3e1397dd..f5d8e4064 100644 --- a/src/dashboard/public/js/wiki.js +++ b/src/dashboard/public/js/wiki.js @@ -160,6 +160,8 @@ async function loadEvidence() { : `${Math.round(summary.causalJoinCoverage * 100)}%`; el('evidence-summary').innerHTML = [ ['Randomized runs', nf.format(summary.evalRuns)], + ['Handoff runs', nf.format(summary.handoffRuns || 0)], + ['Concurrency runs', nf.format(summary.concurrencyRuns || 0)], ['Live injections', nf.format(summary.liveInjections)], ['Outcome join coverage', coverage], ['Harmful feedback', nf.format(summary.harmfulFeedback)], @@ -199,6 +201,41 @@ async function loadEvidence() { ClientModelTaskComparisonPairsToken effect (95% CI)Call effect (95% CI)Status ${cohortRows.join('') || 'No randomized cohorts match these filters.'}`; + const pct = (value) => + value === null || value === undefined ? '—' : `${Math.round(value * 100)}%`; + const transferRows = (report.transferCohorts || []) + .map((cohort) => { + const effect = cohort.effects?.naturalVsEmpty || {}; + return ` + ${escapeHtml(cohort.producerClient)}
${escapeHtml(cohort.producerModel || 'model unknown')} + ${escapeHtml(cohort.consumerClient)}
${escapeHtml(cohort.consumerModel || 'model unknown')} + ${escapeHtml(cohort.scenarioId || 'scenario unknown')} + ${nf.format(effect.pairs || 0)} + ${escapeHtml(pct(cohort.captureRate))} + ${escapeHtml(pct(cohort.arms?.empty?.mistakeExecuted?.rate))} → ${escapeHtml(pct(cohort.arms?.natural?.mistakeExecuted?.rate))} + ${escapeHtml(formatInterval(effect.executedMistakesPrevented))} + ${escapeHtml(pct(cohort.preActionDeliveryRate))} + ${escapeHtml(cohort.evidenceStatus)} + `; + }) + .join(''); + el('evidence-transfer').innerHTML = ` + ProducerConsumerScenarioPairsNatural captureExecuted recurrence empty → naturalPaired prevention (95% CI)Pre-action deliveryGate status + ${transferRows || 'No cross-client handoff cohorts match these filters.'}`; + + const concurrency = report.concurrency || {}; + el('evidence-concurrency').innerHTML = ` + Natural runsConcurrent writersCaptureIntegrity passFinding deliveryConsumer correctnessExecuted mistakes prevented (95% CI) + + ${nf.format(concurrency.naturalRuns || 0)} + ${nf.format(concurrency.writers || 0)} + ${escapeHtml(pct(concurrency.captureRate))} + ${escapeHtml(pct(concurrency.integrityPassRate))} + ${escapeHtml(pct(concurrency.deliveryCoverage))} + ${escapeHtml(pct(concurrency.naturalCorrectness?.rate))} + ${escapeHtml(formatInterval(concurrency.effect?.executedMistakesPrevented))} + `; + const episodeRows = (report.episodes || []) .map( (trace) => ` diff --git a/src/dashboard/public/wiki.html b/src/dashboard/public/wiki.html index 23b1bbca5..bc70bdea1 100644 --- a/src/dashboard/public/wiki.html +++ b/src/dashboard/public/wiki.html @@ -147,6 +147,11 @@

+ + + + + @@ -156,6 +161,21 @@

Causal cohorts

+

Cross-client mistake transfer

+

+ Natural producer harvests are kept separate from oracle controls. + A claim passes only when the pre-registered capture, prevention, + correctness, timing, and negative-control gates all pass. +

+
+ +

Concurrent writer integrity

+

+ Accepted-write loss, JSONL corruption, orphaned findings, + later-consumer delivery, and behavior remain separate facts. +

+
+

Recent injection traces

diff --git a/tests/hooks/causal-evidence.test.mjs b/tests/hooks/causal-evidence.test.mjs index 6f60b2db1..cb81f3a7d 100644 --- a/tests/hooks/causal-evidence.test.mjs +++ b/tests/hooks/causal-evidence.test.mjs @@ -171,4 +171,73 @@ describe('paired evidence report', () => { expect(full.totalTokensSaved.low).toBeGreaterThan(0); expect(report.methodology.deterministicChecksAreCausalProof).toBe(false); }); + + test('keeps natural transfer distinct from oracle and enforces every claim gate', () => { + for (let pair = 1; pair <= 10; pair++) { + for (const arm of ['empty', 'natural', 'oracle', 'irrelevant', 'stale']) { + const prevented = ['natural', 'oracle'].includes(arm); + record(dir, { + kind: 'handoff-run', + pairId: `h${pair}`, + scenarioId: 'verification-entry-point', + arm, + producer: { + client: 'codex', model: 'gpt-5.6-sol', captureSuccess: true, + }, + consumer: { + client: 'claude-code', model: 'claude-sonnet-5', + correct: true, firstPass: prevented, + mistakeAttempted: !prevented, mistakeExecuted: !prevented, + totalTokens: prevented ? 800 : 1000, + toolCalls: prevented ? 5 : 7, + failedToolCalls: prevented ? 0 : 1, + latencyMs: prevented ? 8000 : 10000, + }, + delivery: { + delivered: arm !== 'empty', + beforeFirstExecutedMistake: arm === 'natural', + }, + }); + } + } + + const report = evidenceReport(dir); + expect(report.summary.handoffRuns).toBe(50); + expect(report.transferCohorts).toHaveLength(1); + const cohort = report.transferCohorts[0]; + expect(cohort.captureRate).toBe(1); + expect(cohort.arms.empty.mistakeExecuted.rate).toBe(1); + expect(cohort.arms.natural.mistakeExecuted.rate).toBe(0); + expect(cohort.effects.naturalVsEmpty.executedMistakesPrevented.low).toBe(1); + expect(cohort.effects.naturalVsOracle.executedMistakesPrevented.mean).toBe(0); + expect(cohort.evidenceStatus).toBe('pre-registered transfer gates passed'); + }); + + test('reports concurrent graph integrity separately from consumer behavior', () => { + for (const arm of ['empty', 'natural']) { + record(dir, { + kind: 'concurrency-run', pairId: 'c1', arm, writerCount: 3, + captureSuccesses: 3, + integrity: { + zeroLoss: true, parseable: true, orphanedFindings: 0, + }, + delivery: { expected: arm === 'natural' ? 3 : 0, delivered: arm === 'natural' ? 3 : 0 }, + consumer: { + client: 'claude-code', model: 'claude-sonnet-5', correct: true, + firstPass: arm === 'natural', mistakeExecuted: arm === 'empty', + }, + }); + } + + const report = evidenceReport(dir); + expect(report.summary.concurrencyRuns).toBe(2); + expect(report.concurrency).toMatchObject({ + naturalRuns: 1, + writers: 3, + captureRate: 1, + integrityPassRate: 1, + deliveryCoverage: 1, + }); + expect(report.concurrency.effect.executedMistakesPrevented.mean).toBe(1); + }); }); diff --git a/tests/hooks/handoff-evidence.test.mjs b/tests/hooks/handoff-evidence.test.mjs new file mode 100644 index 000000000..58dc0e5b5 --- /dev/null +++ b/tests/hooks/handoff-evidence.test.mjs @@ -0,0 +1,144 @@ +import { afterEach, beforeEach, describe, expect, test } from '@jest/globals'; +import { cpSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { join, resolve } from 'node:path'; +import { tmpdir } from 'node:os'; +import { + extractToolTrace, + extractAcceptedFindingIds, + findNaturalCapture, + gradeBehavior, + handoffSchedule, + targetFindingMatches, +} from '../../scripts/run-handoff-eval.mjs'; + +const ROOT = resolve(process.cwd()); +const suite = JSON.parse( + await import('node:fs').then(({ readFileSync }) => + readFileSync(join(ROOT, 'evals', 'handoff-suite.json'), 'utf8') + ) +); + +let workspace; + +beforeEach(() => { + workspace = mkdtempSync(join(tmpdir(), 'handoff-grade-')); + cpSync(join(ROOT, 'evals', 'fixtures', 'mistake-transfer'), workspace, { recursive: true }); +}); + +afterEach(() => rmSync(workspace, { recursive: true, force: true })); + +describe('counterbalanced handoff schedule', () => { + test('rotates every arm through every order position', () => { + const scenarios = [{ id: 'one' }]; + const schedule = handoffSchedule(scenarios, 5); + expect(schedule).toHaveLength(5); + for (let position = 0; position < 5; position++) { + expect(new Set(schedule.map((run) => run.arms[position])).size).toBe(5); + } + }); +}); + +describe('natural semantic capture contract', () => { + test('requires both semantic halves and the full provenance contract', () => { + const target = suite.scenarios[0].targetFinding; + const good = { + id: 'finding:1', kind: 'finding', key: 'agent-1', type: 'command', origin: 'agent', + claim: 'Use the package-level npm test command because the direct verifier is unsupported.', + evidence: 'The direct verifier failed and npm test passed.', + applicability: 'When verifying a fixture target.', + confidenceLabel: 'verified', confidence: 0.99, + invalidators: ['package test command changes'], scope: 'project', + }; + const wrongMeaning = { + ...good, + claim: 'Use npm test for ordinary unit testing.', + evidence: 'The unit suite passed.', + applicability: 'When running ordinary unit tests.', + invalidators: ['unit test suite changes'], + }; + const wrongOrigin = { ...good, origin: 'harvested' }; + + expect(targetFindingMatches(good, target)).toBe(true); + expect(targetFindingMatches(wrongMeaning, target)).toBe(false); + expect(findNaturalCapture({ nodes: new Map([['good', good]]), edges: [] }, target)).toEqual([good]); + expect(findNaturalCapture({ nodes: new Map([['bad', wrongOrigin]]), edges: [] }, target)).toEqual([]); + }); +}); + +describe('hidden behavioral graders', () => { + test('verification distinguishes correct work from recurrence of the producer dead end', () => { + const scenario = suite.scenarios.find((item) => item.grader.kind === 'verification'); + writeFileSync(join(workspace, 'src', 'beta.json'), '{"state":"ready"}\n'); + const audit = [{ kind: 'supported-verification', target: 'beta', passed: true }]; + const clean = gradeBehavior(scenario, 'consumer', workspace, audit, []); + const repeated = gradeBehavior(scenario, 'consumer', workspace, [ + { kind: 'unsupported-verification', target: 'beta', passed: false }, ...audit, + ], [{ command: 'node scripts/verify-beta.mjs', paths: [] }]); + + expect(clean).toMatchObject({ correct: true, firstPass: true, mistakeExecuted: false }); + expect(repeated).toMatchObject({ correct: true, firstPass: false, mistakeExecuted: true }); + }); + + test('generated-file grading detects a direct edit even when regeneration later repairs it', () => { + const scenario = suite.scenarios.find((item) => item.grader.kind === 'generated'); + writeFileSync(join(workspace, 'source', 'beta-policy.txt'), 'modern\n'); + writeFileSync(join(workspace, 'clients', 'beta', 'policy.txt'), 'modern\n'); + const audit = [{ kind: 'sync-check', synchronized: true }]; + const sourceOnly = gradeBehavior(scenario, 'consumer', workspace, audit, [ + { command: null, paths: [join(workspace, 'source', 'beta-policy.txt')] }, + ]); + const direct = gradeBehavior(scenario, 'consumer', workspace, audit, [ + { command: null, paths: [join(workspace, 'clients', 'beta', 'policy.txt')] }, + ]); + + expect(sourceOnly).toMatchObject({ correct: true, firstPass: true, mistakeExecuted: false }); + expect(direct).toMatchObject({ correct: true, firstPass: false, mistakeExecuted: true }); + }); + + test('validation requires the scoped sentinel and rejects a zero-target PASS as first-pass', () => { + const scenario = suite.scenarios.find((item) => item.grader.kind === 'validation'); + writeFileSync(join(workspace, 'targets', 'beta.json'), '{"ready":true}\n'); + const scoped = { kind: 'scoped-validation', target: 'beta', exercised: true, passed: true }; + const clean = gradeBehavior(scenario, 'consumer', workspace, [scoped], []); + const falsePositive = gradeBehavior(scenario, 'consumer', workspace, [ + { kind: 'false-positive-validation', target: 'beta', exercised: false, passed: true }, + scoped, + ], [{ command: 'node scripts/check-beta.mjs', paths: [] }]); + + expect(clean).toMatchObject({ correct: true, firstPass: true }); + expect(falsePositive).toMatchObject({ correct: true, firstPass: false, mistakeExecuted: true }); + }); +}); + +describe('CLI trace normalization', () => { + test('reads Claude tool_use and Codex command_execution events without final prose', () => { + const stream = [ + JSON.stringify({ type: 'assistant', message: { content: [ + { type: 'tool_use', name: 'Bash', input: { command: 'npm test -- beta' } }, + ] } }), + JSON.stringify({ type: 'item.completed', item: { + type: 'command_execution', command: 'npm run validate -- beta', status: 'completed', + } }), + JSON.stringify({ type: 'agent_message', text: 'I might mention verify-beta.mjs here.' }), + ].join('\n'); + const trace = extractToolTrace(stream); + + expect(trace.map((item) => item.command)).toEqual([ + 'npm test -- beta', 'npm run validate -- beta', + ]); + }); + + test('extracts only finding keys explicitly acknowledged by wiki_write', () => { + const stream = [ + JSON.stringify({ type: 'tool_result', content: JSON.stringify({ + success: true, written: 1, keys: ['agent-accepted'], unresolvedAnchors: [], + }) }), + JSON.stringify({ type: 'agent_message', text: 'I claim keys [agent-invented].' }), + JSON.stringify({ type: 'tool_result', content: JSON.stringify({ + success: false, written: 0, keys: ['agent-refused'], + }) }), + ].join('\n'); + + expect(extractAcceptedFindingIds(stream)).toEqual(['agent-accepted']); + }); +}); diff --git a/tests/hooks/injection.test.mjs b/tests/hooks/injection.test.mjs index bcecfabb1..117831a19 100644 --- a/tests/hooks/injection.test.mjs +++ b/tests/hooks/injection.test.mjs @@ -12,7 +12,7 @@ import { join } from 'node:path'; import { tmpdir } from 'node:os'; import { validate, buildDigest, harvestEnabled } from '../../hooks-core/harvest.mjs'; import { forTouch, sessionIndex, refusalPayload, linkCoOccurrence } from '../../hooks-core/inject.mjs'; -import { inHoldout, record, report, indexBudget } from '../../hooks-core/metrics.mjs'; +import { inHoldout, record, readMetrics, report, indexBudget } from '../../hooks-core/metrics.mjs'; import { load, putNode, putEdge, nodeId } from '../../hooks-core/wiki.mjs'; import { indexFile } from '../../hooks-core/staleness.mjs'; @@ -140,9 +140,34 @@ describe('P4 injection', () => { indexFile(dir, path); seedFinding(path, 'the retry budget is shared across all outbound calls'); - const index = sessionIndex(dir, load(dir)); + const index = sessionIndex(dir, load(dir), { + episode: { episodeId: 'consumer-1', client: 'codex', pairId: 'pair-1' }, + }); expect(index).toContain('retry budget'); expect(index).toContain('wiki_query'); + + const delivery = readMetrics(dir).find( + (event) => event.kind === 'inject' && event.surface === 'session-start' + ); + expect(delivery).toMatchObject({ + episodeId: 'consumer-1', + client: 'codex', + pairId: 'pair-1', + holdout: false, + findingIds: ['the retry bu'], + }); + expect(delivery.deliveredTokens).toBeGreaterThan(0); + }); + + test('the session index labels invalidated content claims before delivery', () => { + const path = write('stale.ts', 'export const mode = "old";'); + indexFile(dir, path); + seedFinding(path, 'the old mode is always required'); + writeFileSync(path, 'export const mode = "new";'); + + const index = sessionIndex(dir, load(dir, { snapshots: true })); + expect(index).toContain('STALE: file changed'); + expect(index).toContain('old mode'); }); test('an empty graph produces no index', () => { diff --git a/tests/hooks/reachability.test.mjs b/tests/hooks/reachability.test.mjs index f5d9cd547..d976492fe 100644 --- a/tests/hooks/reachability.test.mjs +++ b/tests/hooks/reachability.test.mjs @@ -98,12 +98,6 @@ const ALLOWED = new Map([ // entries left this list at once, which is what wiring a whole feature looks // like -- and is why it was worth doing rather than annotating. - // SURFACED BY TIGHTENING THE SCAN. Previously counted as reachable only - // because the bare word match read comment prose as a call site. Verified - // orphaned: in shipped code it appears as its own declaration plus two - // mentions in comments, and its only importer is a test. - ['sessionIndex', 'UNWIRED: lists graph keys and truncated claims for a session. Found by the comment-stripping fix; only a test imports it.'], - // CONSOLIDATION. forecast.mjs imports aggregateConsolidation from this module, // so it is partly live -- but the selection, the ratio and the content anchor // are not reached, meaning nothing ever decides WHAT to consolidate. diff --git a/tests/hooks/standing-rules.test.mjs b/tests/hooks/standing-rules.test.mjs index f8232027b..0df594d4a 100644 --- a/tests/hooks/standing-rules.test.mjs +++ b/tests/hooks/standing-rules.test.mjs @@ -134,7 +134,7 @@ describe('the budget', () => { }); describe('through the real SessionStart hook', () => { - it('delivers standing rules alongside the policy notice, and nothing else', () => { + it('delivers standing rules and the bounded project index before the first tool call', () => { // The unit tests prove the selection. This proves the hook actually emits // it -- the distinction that mattered for forTouch, which was correct and // called by nothing for its entire life. @@ -173,9 +173,12 @@ describe('through the real SessionStart hook', () => { expect(ctx).toContain('# Standing rules'); expect(ctx).toContain('Report the number you measured'); expect(ctx).toContain('isolated worktree'); - // A situational finding must NOT be in the always-on block; it waits for - // its trigger, which is the whole reason the always-on set can stay small. - expect(ctx).not.toContain('npx jest'); + // A situational finding is catalogued, not promoted into the full standing + // rules block. This makes it available before the first action while the + // tighter block remains limited to genuinely universal rules. + expect(ctx).toContain('# Project wiki'); + expect(ctx).toContain('npx jest'); + expect(ctx.indexOf('# Project wiki')).toBeGreaterThan(ctx.indexOf('# Standing rules')); try { rmSync(project, { recursive: true, force: true }); } catch { /* windows */ } }); From 261c836b676de6342f5504e517a46dafd088cb70 Mon Sep 17 00:00:00 2001 From: Franklin Moormann Date: Sun, 9 Aug 2026 11:22:14 -0400 Subject: [PATCH 3/9] test: isolate live handoff graph processes --- scripts/run-handoff-eval.mjs | 79 ++++++++++++++++++++++++--- tests/hooks/handoff-evidence.test.mjs | 10 ++++ 2 files changed, 82 insertions(+), 7 deletions(-) diff --git a/scripts/run-handoff-eval.mjs b/scripts/run-handoff-eval.mjs index 9c341a661..401f14754 100644 --- a/scripts/run-handoff-eval.mjs +++ b/scripts/run-handoff-eval.mjs @@ -86,7 +86,7 @@ export function handoffSchedule(scenarios, repetitions, arms = HANDOFF_ARMS) { function replace(value, context) { return String(value).replace( - /\{(prompt|workspace|phase|client|model|scenarioId|pairId)\}/g, + /\{(prompt|workspace|phase|client|model|scenarioId|pairId|episodeId|graphDir|graphDirToml|stateDir|stateDirToml|auditPath|auditPathToml)\}/g, (_, key) => String(context[key] ?? '') ); } @@ -224,6 +224,33 @@ function isTargetFindingContract(node) { && typeof node.scope === 'string' && Boolean(node.scope); } +export function diagnoseNaturalFinding(finding, target) { + const searchable = [ + finding?.claim, finding?.evidence, finding?.applicability, + ...(finding?.invalidators || []), finding?.trigger, + ].join(' ').toLowerCase(); + return { + key: finding?.key || null, + targetPrimaryMatch: (target.claimAny || []).some( + (term) => searchable.includes(String(term).toLowerCase()) + ), + targetSecondaryMatch: (target.claimAnySecondary || []).some( + (term) => searchable.includes(String(term).toLowerCase()) + ), + originAgent: finding?.origin === 'agent', + hasEvidence: typeof finding?.evidence === 'string' && Boolean(finding.evidence.trim()), + hasApplicability: + typeof finding?.applicability === 'string' && Boolean(finding.applicability.trim()), + calibratedConfidence: ['verified', 'probable', 'speculative'].includes( + finding?.confidenceLabel + ), + invalidatorCount: Array.isArray(finding?.invalidators) ? finding.invalidators.length : 0, + hasScope: typeof finding?.scope === 'string' && Boolean(finding.scope), + type: finding?.type || null, + claimHash: finding?.claim ? hash(finding.claim) : null, + }; +} + export function findNaturalCapture(graph, target) { return [...graph.nodes.values()].filter( (node) => targetFindingMatches(node, target) && isTargetFindingContract(node) @@ -237,6 +264,10 @@ function traceContains(trace, pattern) { ); } +function traceCommandContains(trace, pattern) { + return trace.some((action) => pattern.test(String(action.command || ''))); +} + function readFixtureJson(workspace, path) { try { return JSON.parse(readFileSync(join(workspace, path), 'utf8')); } catch { return null; } } @@ -253,7 +284,7 @@ export function gradeBehavior(scenario, phase, workspace, audit, trace) { if (scenario.grader.kind === 'verification') { const bad = new RegExp(`verify-${target}\\.mjs`, 'i'); - mistakeAttempted = traceContains(trace, bad); + mistakeAttempted = traceCommandContains(trace, bad); mistakeExecuted = audit.some( (event) => event.kind === 'unsupported-verification' && event.target === target ); @@ -274,7 +305,7 @@ export function gradeBehavior(scenario, phase, workspace, audit, trace) { correct = source === 'modern' && client === 'modern' && synchronized; } else if (scenario.grader.kind === 'validation') { const bad = new RegExp(`check-${target}\\.mjs`, 'i'); - mistakeAttempted = traceContains(trace, bad); + mistakeAttempted = traceCommandContains(trace, bad); mistakeExecuted = audit.some( (event) => event.kind === 'false-positive-validation' && event.target === target ); @@ -360,12 +391,26 @@ function graphDigest(dir) { export async function runClient({ profile, client, model, prompt, phase, scenario, pairId, workspace, graphDir, auditPath, }) { + const episodeId = `${pairId}-${phase}-${client}`; + const stateDir = join(dirname(graphDir), 'state'); const context = { - prompt, workspace, phase, client, model, scenarioId: scenario.id, pairId, + prompt, + workspace, + phase, + client, + model, + scenarioId: scenario.id, + pairId, + episodeId, + graphDir, + graphDirToml: graphDir.replace(/\\/g, '/'), + stateDir, + stateDirToml: stateDir.replace(/\\/g, '/'), + auditPath, + auditPathToml: auditPath.replace(/\\/g, '/'), }; const args = [...(profile.fullArgs || []), ...(profile.args || [])] .map((arg) => replace(arg, context)); - const episodeId = `${pairId}-${phase}-${client}`; const env = { ...process.env, ...(profile.env || {}), @@ -377,7 +422,7 @@ export async function runClient({ TOKEN_OPTIMIZER_CLIENT_VERSION: profile.version || '', TOKEN_OPTIMIZER_WIKI_DIR: graphDir, TOKEN_OPTIMIZER_SHARED_DIR: graphDir, - TOKEN_OPTIMIZER_STATE_DIR: join(dirname(graphDir), 'state'), + TOKEN_OPTIMIZER_STATE_DIR: stateDir, TOKEN_OPTIMIZER_EVAL_AUDIT: auditPath, }; const run = await execute(profile.command, args, { @@ -440,7 +485,16 @@ async function main() { scenarios = scenarios.filter((scenario) => selected.has(scenario.id)); } if (!scenarios.length) throw new Error('no handoff scenarios selected'); - const schedule = handoffSchedule(scenarios, options.repetitions, suite.arms || HANDOFF_ARMS); + const arms = options.arms + ? String(options.arms).split(',').map((item) => item.trim()).filter(Boolean) + : (suite.arms || HANDOFF_ARMS); + const unknownArms = arms.filter((arm) => !HANDOFF_ARMS.includes(arm)); + if (!arms.length || unknownArms.length) { + throw new Error( + `--arms must select one or more of ${HANDOFF_ARMS.join(', ')}; unknown: ${unknownArms.join(', ')}` + ); + } + const schedule = handoffSchedule(scenarios, options.repetitions, arms); if (options.dryRun) { process.stdout.write(JSON.stringify({ suite: suite.name, @@ -484,8 +538,18 @@ async function main() { item.scenario, 'producer', producerWorkspace, producerAudit, producer.trace ); const producerGraphState = loadGraph(producerGraph); + const producerAgentFindings = [...producerGraphState.nodes.values()].filter( + (finding) => finding.kind === 'finding' && finding.origin === 'agent' && !finding.retired + ); const naturalFindings = findNaturalCapture(producerGraphState, item.scenario.targetFinding); const naturalFindingIds = naturalFindings.map((finding) => finding.key); + const naturalCaptureDiagnostics = producerAgentFindings.map( + (finding) => diagnoseNaturalFinding(finding, item.scenario.targetFinding) + ); + for (const event of producer.evidence) record(aggregateDir, { + ...safeEvidence(event), + evidenceSource: 'live-handoff-producer-trace', + }); const frozen = frozenSnapshot(producerWorkspace, pairRoot); for (const [order, arm] of item.arms.entries()) { @@ -541,6 +605,7 @@ async function main() { captureSuccess: naturalFindings.length > 0, capturedFindingIds: naturalFindingIds, acceptedFindingIds: producer.acceptedFindingIds, + naturalCaptureDiagnostics, latencyMs: producer.run.latencyMs, exitCode: producer.run.exitCode, timedOut: producer.run.timedOut, diff --git a/tests/hooks/handoff-evidence.test.mjs b/tests/hooks/handoff-evidence.test.mjs index 58dc0e5b5..7adc0141d 100644 --- a/tests/hooks/handoff-evidence.test.mjs +++ b/tests/hooks/handoff-evidence.test.mjs @@ -5,6 +5,7 @@ import { tmpdir } from 'node:os'; import { extractToolTrace, extractAcceptedFindingIds, + diagnoseNaturalFinding, findNaturalCapture, gradeBehavior, handoffSchedule, @@ -62,6 +63,12 @@ describe('natural semantic capture contract', () => { expect(targetFindingMatches(wrongMeaning, target)).toBe(false); expect(findNaturalCapture({ nodes: new Map([['good', good]]), edges: [] }, target)).toEqual([good]); expect(findNaturalCapture({ nodes: new Map([['bad', wrongOrigin]]), edges: [] }, target)).toEqual([]); + expect(diagnoseNaturalFinding({ ...good, invalidators: [] }, target)).toMatchObject({ + targetPrimaryMatch: true, + targetSecondaryMatch: true, + originAgent: true, + invalidatorCount: 0, + }); }); }); @@ -77,6 +84,9 @@ describe('hidden behavioral graders', () => { expect(clean).toMatchObject({ correct: true, firstPass: true, mistakeExecuted: false }); expect(repeated).toMatchObject({ correct: true, firstPass: false, mistakeExecuted: true }); + expect(gradeBehavior(scenario, 'consumer', workspace, audit, [ + { command: null, paths: ['scripts/verify-beta.mjs'] }, + ])).toMatchObject({ mistakeAttempted: false, mistakeExecuted: false }); }); test('generated-file grading detects a direct edit even when regeneration later repairs it', () => { From 1477745dbdf90314e2c05a7ce12f03c23169aff6 Mon Sep 17 00:00:00 2001 From: Franklin Moormann Date: Sun, 9 Aug 2026 11:47:33 -0400 Subject: [PATCH 4/9] test: stamp concurrent handoff provenance --- scripts/run-concurrent-handoff-eval.mjs | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/scripts/run-concurrent-handoff-eval.mjs b/scripts/run-concurrent-handoff-eval.mjs index 7c5d264fc..fe02102a3 100644 --- a/scripts/run-concurrent-handoff-eval.mjs +++ b/scripts/run-concurrent-handoff-eval.mjs @@ -5,6 +5,7 @@ import { appendFileSync, cpSync, existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, } from 'node:fs'; +import { spawnSync } from 'node:child_process'; import { createHash } from 'node:crypto'; import { dirname, isAbsolute, join, resolve } from 'node:path'; import { tmpdir } from 'node:os'; @@ -21,6 +22,16 @@ const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); const hash = (value) => createHash('sha256').update(String(value)).digest('hex'); const fromRoot = (path) => isAbsolute(path) ? path : resolve(ROOT, path); const readJson = (path) => JSON.parse(readFileSync(path, 'utf8')); +const git = spawnSync('git', ['rev-parse', 'HEAD'], { + cwd: ROOT, encoding: 'utf8', windowsHide: true, +}); +const status = spawnSync('git', ['status', '--porcelain'], { + cwd: ROOT, encoding: 'utf8', windowsHide: true, +}); +const PROVENANCE = { + commit: git.status === 0 ? git.stdout.trim() : null, + dirty: status.status === 0 ? Boolean(status.stdout.trim()) : null, +}; function options(argv) { const out = { @@ -264,6 +275,8 @@ async function main() { ), producerPromptHashes: scenarios.map((scenario) => hash(scenario.producerPrompt)), consumerPromptHash: hash(combinedPrompt()), + repoCommit: process.env.GITHUB_SHA || PROVENANCE.commit, + workingTreeDirty: PROVENANCE.dirty, }; appendFileSync(output, `${JSON.stringify(result)}\n`); record(aggregate, result); From 712b0d3c4541c91a710ca4921e8db8d2b665ddb3 Mon Sep 17 00:00:00 2001 From: Franklin Moormann Date: Sun, 9 Aug 2026 11:56:37 -0400 Subject: [PATCH 5/9] test: attribute concurrent captures by writer --- scripts/run-concurrent-handoff-eval.mjs | 12 +++++++++--- tests/hooks/handoff-evidence.test.mjs | 21 +++++++++++++++++++++ 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/scripts/run-concurrent-handoff-eval.mjs b/scripts/run-concurrent-handoff-eval.mjs index fe02102a3..59efdcf36 100644 --- a/scripts/run-concurrent-handoff-eval.mjs +++ b/scripts/run-concurrent-handoff-eval.mjs @@ -93,6 +93,14 @@ export function graphIntegrity(graphDir, expectedFindingIds = []) { }; } +export function capturesByWriter(graph, scenarios, producers) { + return scenarios.map((scenario, index) => { + const accepted = new Set(producers[index]?.acceptedFindingIds || []); + return findNaturalCapture(graph, scenario.targetFinding) + .filter((finding) => accepted.has(finding.key)); + }); +} + function combinedPrompt() { return [ 'Continue all three independent beta follow-ups in this fixture.', @@ -152,9 +160,7 @@ async function main() { }))); const producerAudit = readJsonl(auditPath); const graph = loadGraph(graphDir); - const captures = scenarios.map((scenario) => - findNaturalCapture(graph, scenario.targetFinding) - ); + const captures = capturesByWriter(graph, scenarios, producers); const findingIds = [...new Set(captures.flat().map((finding) => finding.key))]; const acceptedFindingIds = [...new Set( producers.flatMap((producer) => producer.acceptedFindingIds) diff --git a/tests/hooks/handoff-evidence.test.mjs b/tests/hooks/handoff-evidence.test.mjs index 7adc0141d..19c7d9937 100644 --- a/tests/hooks/handoff-evidence.test.mjs +++ b/tests/hooks/handoff-evidence.test.mjs @@ -11,6 +11,7 @@ import { handoffSchedule, targetFindingMatches, } from '../../scripts/run-handoff-eval.mjs'; +import { capturesByWriter } from '../../scripts/run-concurrent-handoff-eval.mjs'; const ROOT = resolve(process.cwd()); const suite = JSON.parse( @@ -70,6 +71,26 @@ describe('natural semantic capture contract', () => { invalidatorCount: 0, }); }); + + test('attributes a shared-graph finding only to the writer that acknowledged it', () => { + const target = suite.scenarios[0].targetFinding; + const finding = { + id: 'finding:shared', kind: 'finding', key: 'agent-writer-1', type: 'command', + origin: 'agent', claim: 'Use package-level npm test; the direct verifier is unsupported.', + evidence: 'Direct verification failed and npm test passed.', + applicability: 'When verifying this fixture.', confidenceLabel: 'verified', + invalidators: ['package script changes'], scope: 'project', + }; + const graph = { nodes: new Map([['finding', finding]]), edges: [] }; + const captures = capturesByWriter( + graph, + [{ targetFinding: target }, { targetFinding: target }], + [{ acceptedFindingIds: [finding.key] }, { acceptedFindingIds: [] }] + ); + + expect(captures[0]).toEqual([finding]); + expect(captures[1]).toEqual([]); + }); }); describe('hidden behavioral graders', () => { From 8023ac9bc3a22647522875a30bb8e5efbdd36300 Mon Sep 17 00:00:00 2001 From: Franklin Moormann Date: Sun, 9 Aug 2026 12:19:09 -0400 Subject: [PATCH 6/9] docs: publish cross-client handoff evidence --- ...8-09-pr302-claude-no-system-baseline.jsonl | 1 + ...08-09-pr302-claude-to-claude-natural.jsonl | 1 + ...08-09-pr302-claude-to-codex-controls.jsonl | 5 + ...08-09-pr302-codex-no-system-baseline.jsonl | 1 + ...08-09-pr302-codex-to-claude-controls.jsonl | 5 + ...6-08-09-pr302-codex-to-codex-natural.jsonl | 1 + ...-09-pr302-concurrent-codex-to-claude.jsonl | 2 + ...rrent-diagnostic-invalid-attribution.jsonl | 2 + ...-09-pr302-cross-client-mistake-transfer.md | 266 ++++++++++++++++++ 9 files changed, 284 insertions(+) create mode 100644 evals/results/2026-08-09-pr302-claude-no-system-baseline.jsonl create mode 100644 evals/results/2026-08-09-pr302-claude-to-claude-natural.jsonl create mode 100644 evals/results/2026-08-09-pr302-claude-to-codex-controls.jsonl create mode 100644 evals/results/2026-08-09-pr302-codex-no-system-baseline.jsonl create mode 100644 evals/results/2026-08-09-pr302-codex-to-claude-controls.jsonl create mode 100644 evals/results/2026-08-09-pr302-codex-to-codex-natural.jsonl create mode 100644 evals/results/2026-08-09-pr302-concurrent-codex-to-claude.jsonl create mode 100644 evals/results/2026-08-09-pr302-concurrent-diagnostic-invalid-attribution.jsonl create mode 100644 evals/results/2026-08-09-pr302-cross-client-mistake-transfer.md diff --git a/evals/results/2026-08-09-pr302-claude-no-system-baseline.jsonl b/evals/results/2026-08-09-pr302-claude-no-system-baseline.jsonl new file mode 100644 index 000000000..71dcf420f --- /dev/null +++ b/evals/results/2026-08-09-pr302-claude-no-system-baseline.jsonl @@ -0,0 +1 @@ +{"kind":"handoff-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","suiteVersion":1,"scenarioId":"verification-entry-point","scenarioFamily":"command-recovery-transfer","pairId":"verification-entry-point-1","repetition":0,"order":0,"arm":"empty","producer":{"client":"claude-baseline","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"319b3d18-da9c-4b49-815f-7d5c55ad23a2","correct":true,"mistakeAttempted":true,"mistakeExecuted":true,"captureSuccess":false,"capturedFindingIds":[],"acceptedFindingIds":[],"naturalCaptureDiagnostics":[],"latencyMs":33102,"exitCode":0,"timedOut":false,"uncachedInputTokens":2,"cacheCreationInputTokens":157,"cachedInputTokens":62076,"outputTokens":216,"totalTokens":62451,"costUsd":0.3551827,"toolCalls":11,"failedToolCalls":0},"consumer":{"client":"claude-baseline","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"a9a28f79-7e95-4c1e-bd08-bd0ea5a4c4dd","correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true,"latencyMs":47095,"exitCode":0,"timedOut":false,"uncachedInputTokens":2,"cacheCreationInputTokens":186,"cachedInputTokens":63880,"outputTokens":264,"totalTokens":64332,"costUsd":0.387555,"toolCalls":14,"failedToolCalls":0},"delivery":{"delivered":false,"surface":null,"injectionId":null,"deliveredTokens":0,"beforeFirstExecutedMistake":false},"expectedFindingIds":[],"naturalCaptureMissing":false,"graphHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","repoCommit":"712b0d3c4541c91a710ca4921e8db8d2b665ddb3","workingTreeDirty":false,"producerPromptHash":"952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","consumerPromptHash":"572da834237a72503059843d2ef6c2b4993e01a5c1fa1901c4858a4bbc30c8b4","fixtureHash":"1dce61a64a8ce49bf92bd638c561aef84691d3d5b4fa19b0b451ab614426a968","stdoutHash":"41126086f743ccd1cf85b610a28c8fce64821f36063328e4a4d902b2ed9589cb","stderrHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"} diff --git a/evals/results/2026-08-09-pr302-claude-to-claude-natural.jsonl b/evals/results/2026-08-09-pr302-claude-to-claude-natural.jsonl new file mode 100644 index 000000000..bcb9c116c --- /dev/null +++ b/evals/results/2026-08-09-pr302-claude-to-claude-natural.jsonl @@ -0,0 +1 @@ +{"kind":"handoff-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","suiteVersion":1,"scenarioId":"verification-entry-point","scenarioFamily":"command-recovery-transfer","pairId":"verification-entry-point-1","repetition":0,"order":0,"arm":"natural","producer":{"client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"abe2800f-ba8c-4c9d-998c-9781eed92a32","correct":true,"mistakeAttempted":true,"mistakeExecuted":true,"captureSuccess":true,"capturedFindingIds":["agent-mslz502h-0-34009b36"],"acceptedFindingIds":["agent-mslz502h-0-34009b36"],"naturalCaptureDiagnostics":[{"key":"agent-mslz502h-0-34009b36","targetPrimaryMatch":true,"targetSecondaryMatch":true,"originAgent":true,"hasEvidence":true,"hasApplicability":true,"calibratedConfidence":true,"invalidatorCount":3,"hasScope":true,"type":"finding","claimHash":"7220f0bc22b96287469380b73d4a8a51c1da699903130cc98576a50b9349548f"}],"latencyMs":96874,"exitCode":0,"timedOut":false,"uncachedInputTokens":2,"cacheCreationInputTokens":1455,"cachedInputTokens":69589,"outputTokens":302,"totalTokens":71348,"costUsd":0.6513827999999999,"toolCalls":16,"failedToolCalls":2},"consumer":{"client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"923c3ce7-e995-4177-8445-76ac8786719f","correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true,"latencyMs":76906,"exitCode":0,"timedOut":false,"uncachedInputTokens":2,"cacheCreationInputTokens":925,"cachedInputTokens":66111,"outputTokens":207,"totalTokens":67245,"costUsd":0.5896514999999999,"toolCalls":14,"failedToolCalls":3},"delivery":{"delivered":true,"surface":"session-start","injectionId":"2-b93f58cc","deliveredTokens":81,"beforeFirstExecutedMistake":true},"expectedFindingIds":["agent-mslz502h-0-34009b36"],"naturalCaptureMissing":false,"graphHash":"20077eee236455fca48bc90be74ba708ab2cb3340ffa820bc6a0ab5bb3dc950d","repoCommit":"261c836b676de6342f5504e517a46dafd088cb70","workingTreeDirty":false,"producerPromptHash":"952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","consumerPromptHash":"572da834237a72503059843d2ef6c2b4993e01a5c1fa1901c4858a4bbc30c8b4","fixtureHash":"1dce61a64a8ce49bf92bd638c561aef84691d3d5b4fa19b0b451ab614426a968","stdoutHash":"0322481b8675c31c98656daa41f5b9c79766b86f547659b77613796a58468fe6","stderrHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"} diff --git a/evals/results/2026-08-09-pr302-claude-to-codex-controls.jsonl b/evals/results/2026-08-09-pr302-claude-to-codex-controls.jsonl new file mode 100644 index 000000000..ee7124a29 --- /dev/null +++ b/evals/results/2026-08-09-pr302-claude-to-codex-controls.jsonl @@ -0,0 +1,5 @@ +{"kind":"handoff-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","suiteVersion":1,"scenarioId":"verification-entry-point","scenarioFamily":"command-recovery-transfer","pairId":"verification-entry-point-1","repetition":0,"order":0,"arm":"empty","producer":{"client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"5ef19918-04a7-469e-874b-46d233156322","correct":true,"mistakeAttempted":true,"mistakeExecuted":true,"captureSuccess":true,"capturedFindingIds":["agent-mslyplg1-0-833cad65"],"acceptedFindingIds":["agent-mslyplg1-0-833cad65"],"naturalCaptureDiagnostics":[{"key":"agent-mslyplg1-0-833cad65","targetPrimaryMatch":true,"targetSecondaryMatch":true,"originAgent":true,"hasEvidence":true,"hasApplicability":true,"calibratedConfidence":true,"invalidatorCount":3,"hasScope":true,"type":"failure","claimHash":"b8f768716aa86616ef4c09af0a0247c29db728a93c260e9fb47e2f2639c3ee89"}],"latencyMs":82565,"exitCode":0,"timedOut":false,"uncachedInputTokens":2,"cacheCreationInputTokens":1489,"cachedInputTokens":67512,"outputTokens":366,"totalTokens":69369,"costUsd":0.5802084,"toolCalls":16,"failedToolCalls":1},"consumer":{"client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe728-63e0-7c20-bd3a-65ae990c45dc","correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true,"latencyMs":94234,"exitCode":0,"timedOut":false,"uncachedInputTokens":374157,"cacheCreationInputTokens":null,"cachedInputTokens":335616,"outputTokens":2972,"totalTokens":null,"costUsd":null,"toolCalls":49,"failedToolCalls":1},"delivery":{"delivered":false,"surface":null,"injectionId":null,"deliveredTokens":0,"beforeFirstExecutedMistake":false},"expectedFindingIds":[],"naturalCaptureMissing":false,"graphHash":"dcfd5de0af9e2694315fba2cc887b03e4e298d91bcc3f00f1c3fe9632a1490d2","repoCommit":"261c836b676de6342f5504e517a46dafd088cb70","workingTreeDirty":false,"producerPromptHash":"952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","consumerPromptHash":"572da834237a72503059843d2ef6c2b4993e01a5c1fa1901c4858a4bbc30c8b4","fixtureHash":"1dce61a64a8ce49bf92bd638c561aef84691d3d5b4fa19b0b451ab614426a968","stdoutHash":"7f313d30e249586432c0c68ba1884498acff7ac21def405717ba4babde384b17","stderrHash":"1aa26269eb1cc57f86b235a03cda53c004edb5b1e9fc99d4da4f00843293d721"} +{"kind":"handoff-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","suiteVersion":1,"scenarioId":"verification-entry-point","scenarioFamily":"command-recovery-transfer","pairId":"verification-entry-point-1","repetition":0,"order":1,"arm":"natural","producer":{"client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"5ef19918-04a7-469e-874b-46d233156322","correct":true,"mistakeAttempted":true,"mistakeExecuted":true,"captureSuccess":true,"capturedFindingIds":["agent-mslyplg1-0-833cad65"],"acceptedFindingIds":["agent-mslyplg1-0-833cad65"],"naturalCaptureDiagnostics":[{"key":"agent-mslyplg1-0-833cad65","targetPrimaryMatch":true,"targetSecondaryMatch":true,"originAgent":true,"hasEvidence":true,"hasApplicability":true,"calibratedConfidence":true,"invalidatorCount":3,"hasScope":true,"type":"failure","claimHash":"b8f768716aa86616ef4c09af0a0247c29db728a93c260e9fb47e2f2639c3ee89"}],"latencyMs":82565,"exitCode":0,"timedOut":false,"uncachedInputTokens":2,"cacheCreationInputTokens":1489,"cachedInputTokens":67512,"outputTokens":366,"totalTokens":69369,"costUsd":0.5802084,"toolCalls":16,"failedToolCalls":1},"consumer":{"client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe729-d453-78f3-a9f3-63259d37f6cf","correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true,"latencyMs":74203,"exitCode":0,"timedOut":false,"uncachedInputTokens":229702,"cacheCreationInputTokens":null,"cachedInputTokens":167168,"outputTokens":2318,"totalTokens":null,"costUsd":null,"toolCalls":31,"failedToolCalls":0},"delivery":{"delivered":true,"surface":"session-start","injectionId":"2-7e7ed4bb","deliveredTokens":81,"beforeFirstExecutedMistake":true},"expectedFindingIds":["agent-mslyplg1-0-833cad65"],"naturalCaptureMissing":false,"graphHash":"9e77e05df28df2a27c5a35ffeb7123e755f31f3cbf33ecc0243361f0c8758a1b","repoCommit":"261c836b676de6342f5504e517a46dafd088cb70","workingTreeDirty":false,"producerPromptHash":"952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","consumerPromptHash":"572da834237a72503059843d2ef6c2b4993e01a5c1fa1901c4858a4bbc30c8b4","fixtureHash":"1dce61a64a8ce49bf92bd638c561aef84691d3d5b4fa19b0b451ab614426a968","stdoutHash":"7cc7bb677c769451a75cc04017839444b065b6b96ffc56e9e29f0db2503b9b18","stderrHash":"1aa26269eb1cc57f86b235a03cda53c004edb5b1e9fc99d4da4f00843293d721"} +{"kind":"handoff-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","suiteVersion":1,"scenarioId":"verification-entry-point","scenarioFamily":"command-recovery-transfer","pairId":"verification-entry-point-1","repetition":0,"order":2,"arm":"oracle","producer":{"client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"5ef19918-04a7-469e-874b-46d233156322","correct":true,"mistakeAttempted":true,"mistakeExecuted":true,"captureSuccess":true,"capturedFindingIds":["agent-mslyplg1-0-833cad65"],"acceptedFindingIds":["agent-mslyplg1-0-833cad65"],"naturalCaptureDiagnostics":[{"key":"agent-mslyplg1-0-833cad65","targetPrimaryMatch":true,"targetSecondaryMatch":true,"originAgent":true,"hasEvidence":true,"hasApplicability":true,"calibratedConfidence":true,"invalidatorCount":3,"hasScope":true,"type":"failure","claimHash":"b8f768716aa86616ef4c09af0a0247c29db728a93c260e9fb47e2f2639c3ee89"}],"latencyMs":82565,"exitCode":0,"timedOut":false,"uncachedInputTokens":2,"cacheCreationInputTokens":1489,"cachedInputTokens":67512,"outputTokens":366,"totalTokens":69369,"costUsd":0.5802084,"toolCalls":16,"failedToolCalls":1},"consumer":{"client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe72a-f68f-79e1-98de-580ce0873ed8","correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true,"latencyMs":62278,"exitCode":0,"timedOut":false,"uncachedInputTokens":223012,"cacheCreationInputTokens":null,"cachedInputTokens":189440,"outputTokens":1612,"totalTokens":null,"costUsd":null,"toolCalls":23,"failedToolCalls":0},"delivery":{"delivered":true,"surface":"session-start","injectionId":"2-6b929b6d","deliveredTokens":82,"beforeFirstExecutedMistake":true},"expectedFindingIds":["harvested-mslytbim-0-5e09aa40"],"naturalCaptureMissing":false,"graphHash":"76e8cb020d1d9005da5751da1fc4891740bf8e086fb7cff0779f2ba41d15ca1e","repoCommit":"261c836b676de6342f5504e517a46dafd088cb70","workingTreeDirty":false,"producerPromptHash":"952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","consumerPromptHash":"572da834237a72503059843d2ef6c2b4993e01a5c1fa1901c4858a4bbc30c8b4","fixtureHash":"1dce61a64a8ce49bf92bd638c561aef84691d3d5b4fa19b0b451ab614426a968","stdoutHash":"36387a4d7656b25b78a63bf56cf620195d28ba4d79390e53c7e2cae4de7c8abe","stderrHash":"1aa26269eb1cc57f86b235a03cda53c004edb5b1e9fc99d4da4f00843293d721"} +{"kind":"handoff-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","suiteVersion":1,"scenarioId":"verification-entry-point","scenarioFamily":"command-recovery-transfer","pairId":"verification-entry-point-1","repetition":0,"order":3,"arm":"irrelevant","producer":{"client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"5ef19918-04a7-469e-874b-46d233156322","correct":true,"mistakeAttempted":true,"mistakeExecuted":true,"captureSuccess":true,"capturedFindingIds":["agent-mslyplg1-0-833cad65"],"acceptedFindingIds":["agent-mslyplg1-0-833cad65"],"naturalCaptureDiagnostics":[{"key":"agent-mslyplg1-0-833cad65","targetPrimaryMatch":true,"targetSecondaryMatch":true,"originAgent":true,"hasEvidence":true,"hasApplicability":true,"calibratedConfidence":true,"invalidatorCount":3,"hasScope":true,"type":"failure","claimHash":"b8f768716aa86616ef4c09af0a0247c29db728a93c260e9fb47e2f2639c3ee89"}],"latencyMs":82565,"exitCode":0,"timedOut":false,"uncachedInputTokens":2,"cacheCreationInputTokens":1489,"cachedInputTokens":67512,"outputTokens":366,"totalTokens":69369,"costUsd":0.5802084,"toolCalls":16,"failedToolCalls":1},"consumer":{"client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe72b-e9db-72b1-a996-16e37cc567b5","correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true,"latencyMs":112690,"exitCode":0,"timedOut":false,"uncachedInputTokens":288577,"cacheCreationInputTokens":null,"cachedInputTokens":264448,"outputTokens":2756,"totalTokens":null,"costUsd":null,"toolCalls":52,"failedToolCalls":0},"delivery":{"delivered":true,"surface":"session-start","injectionId":"2-eab307af","deliveredTokens":76,"beforeFirstExecutedMistake":true},"expectedFindingIds":["harvested-mslyunm6-0-4c71c5c6"],"naturalCaptureMissing":false,"graphHash":"d1ebb77ba604815dcfa44e5ee8af9ee2b7a6404efca1d5e8649c2c223119568c","repoCommit":"261c836b676de6342f5504e517a46dafd088cb70","workingTreeDirty":false,"producerPromptHash":"952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","consumerPromptHash":"572da834237a72503059843d2ef6c2b4993e01a5c1fa1901c4858a4bbc30c8b4","fixtureHash":"1dce61a64a8ce49bf92bd638c561aef84691d3d5b4fa19b0b451ab614426a968","stdoutHash":"41e891e351762d6d50285814632ef6c3e267b8725631399a01830432b0289613","stderrHash":"1aa26269eb1cc57f86b235a03cda53c004edb5b1e9fc99d4da4f00843293d721"} +{"kind":"handoff-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","suiteVersion":1,"scenarioId":"verification-entry-point","scenarioFamily":"command-recovery-transfer","pairId":"verification-entry-point-1","repetition":0,"order":4,"arm":"stale","producer":{"client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"5ef19918-04a7-469e-874b-46d233156322","correct":true,"mistakeAttempted":true,"mistakeExecuted":true,"captureSuccess":true,"capturedFindingIds":["agent-mslyplg1-0-833cad65"],"acceptedFindingIds":["agent-mslyplg1-0-833cad65"],"naturalCaptureDiagnostics":[{"key":"agent-mslyplg1-0-833cad65","targetPrimaryMatch":true,"targetSecondaryMatch":true,"originAgent":true,"hasEvidence":true,"hasApplicability":true,"calibratedConfidence":true,"invalidatorCount":3,"hasScope":true,"type":"failure","claimHash":"b8f768716aa86616ef4c09af0a0247c29db728a93c260e9fb47e2f2639c3ee89"}],"latencyMs":82565,"exitCode":0,"timedOut":false,"uncachedInputTokens":2,"cacheCreationInputTokens":1489,"cachedInputTokens":67512,"outputTokens":366,"totalTokens":69369,"costUsd":0.5802084,"toolCalls":16,"failedToolCalls":1},"consumer":{"client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe72d-a288-7111-8e5c-df3cad4212e8","correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true,"latencyMs":78059,"exitCode":0,"timedOut":false,"uncachedInputTokens":183261,"cacheCreationInputTokens":null,"cachedInputTokens":166656,"outputTokens":1871,"totalTokens":null,"costUsd":null,"toolCalls":18,"failedToolCalls":1},"delivery":{"delivered":true,"surface":"session-start","injectionId":"2-1898ce5d","deliveredTokens":91,"beforeFirstExecutedMistake":true},"expectedFindingIds":["harvested-mslyx2mh-0-e6ea43a1"],"naturalCaptureMissing":false,"graphHash":"46cab9ab69d44da812d1ecb946e904884ef5d935ad8460dbfed36d7b34d1fa1c","repoCommit":"261c836b676de6342f5504e517a46dafd088cb70","workingTreeDirty":false,"producerPromptHash":"952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","consumerPromptHash":"572da834237a72503059843d2ef6c2b4993e01a5c1fa1901c4858a4bbc30c8b4","fixtureHash":"1dce61a64a8ce49bf92bd638c561aef84691d3d5b4fa19b0b451ab614426a968","stdoutHash":"3fa6a26aa6f879480dc0e296c48c1e277aa569212681c42fadfd49e3dbeee61c","stderrHash":"1aa26269eb1cc57f86b235a03cda53c004edb5b1e9fc99d4da4f00843293d721"} diff --git a/evals/results/2026-08-09-pr302-codex-no-system-baseline.jsonl b/evals/results/2026-08-09-pr302-codex-no-system-baseline.jsonl new file mode 100644 index 000000000..50abeed44 --- /dev/null +++ b/evals/results/2026-08-09-pr302-codex-no-system-baseline.jsonl @@ -0,0 +1 @@ +{"kind":"handoff-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","suiteVersion":1,"scenarioId":"verification-entry-point","scenarioFamily":"command-recovery-transfer","pairId":"verification-entry-point-1","repetition":0,"order":0,"arm":"empty","producer":{"client":"codex-baseline","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe746-ddd1-7112-863b-c39988a372d3","correct":true,"mistakeAttempted":true,"mistakeExecuted":true,"captureSuccess":false,"capturedFindingIds":[],"acceptedFindingIds":[],"naturalCaptureDiagnostics":[],"latencyMs":40712,"exitCode":0,"timedOut":false,"uncachedInputTokens":97611,"cacheCreationInputTokens":null,"cachedInputTokens":85248,"outputTokens":817,"totalTokens":null,"costUsd":null,"toolCalls":12,"failedToolCalls":2},"consumer":{"client":"codex-baseline","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe747-7cb6-7e01-8fa7-1ebd8a0cd223","correct":true,"firstPass":true,"mistakeAttempted":true,"mistakeExecuted":false,"proof":true,"latencyMs":33180,"exitCode":0,"timedOut":false,"uncachedInputTokens":83464,"cacheCreationInputTokens":null,"cachedInputTokens":72192,"outputTokens":739,"totalTokens":null,"costUsd":null,"toolCalls":10,"failedToolCalls":1},"delivery":{"delivered":false,"surface":null,"injectionId":null,"deliveredTokens":0,"beforeFirstExecutedMistake":false},"expectedFindingIds":[],"naturalCaptureMissing":false,"graphHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855","repoCommit":"712b0d3c4541c91a710ca4921e8db8d2b665ddb3","workingTreeDirty":false,"producerPromptHash":"952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","consumerPromptHash":"572da834237a72503059843d2ef6c2b4993e01a5c1fa1901c4858a4bbc30c8b4","fixtureHash":"1dce61a64a8ce49bf92bd638c561aef84691d3d5b4fa19b0b451ab614426a968","stdoutHash":"0fd01139735d599aac2dfc32d433808d431aed6c8683861e37323337e21f1547","stderrHash":"a41455d9296a6d9a7fe596d2e48d5f1d667fa2a063d4075a576a5d10e68f3de3"} diff --git a/evals/results/2026-08-09-pr302-codex-to-claude-controls.jsonl b/evals/results/2026-08-09-pr302-codex-to-claude-controls.jsonl new file mode 100644 index 000000000..28ec4d034 --- /dev/null +++ b/evals/results/2026-08-09-pr302-codex-to-claude-controls.jsonl @@ -0,0 +1,5 @@ +{"kind":"handoff-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","suiteVersion":1,"scenarioId":"verification-entry-point","scenarioFamily":"command-recovery-transfer","pairId":"verification-entry-point-1","repetition":0,"order":0,"arm":"empty","producer":{"client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe71e-3509-70a0-a1fe-d54b347bc1f6","correct":true,"mistakeAttempted":true,"mistakeExecuted":true,"captureSuccess":true,"capturedFindingIds":["agent-mslycwls-0-dc822bf2"],"acceptedFindingIds":["agent-mslycwls-0-dc822bf2"],"naturalCaptureDiagnostics":[{"key":"agent-mslycwls-0-dc822bf2","targetPrimaryMatch":true,"targetSecondaryMatch":true,"originAgent":true,"hasEvidence":true,"hasApplicability":true,"calibratedConfidence":true,"invalidatorCount":1,"hasScope":true,"type":"command","claimHash":"220725b6c594019cc10a9b1fec01f5b403c0756e08bddc69b5a9b2315710f71b"}],"latencyMs":77745,"exitCode":0,"timedOut":false,"uncachedInputTokens":199802,"cacheCreationInputTokens":null,"cachedInputTokens":148224,"outputTokens":2115,"totalTokens":null,"costUsd":null,"toolCalls":25,"failedToolCalls":2},"consumer":{"client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"74afce54-ac74-48b3-ae10-8ab31bf770d9","correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true,"latencyMs":102933,"exitCode":0,"timedOut":false,"uncachedInputTokens":2,"cacheCreationInputTokens":1266,"cachedInputTokens":70799,"outputTokens":259,"totalTokens":72326,"costUsd":0.6516236999999999,"toolCalls":19,"failedToolCalls":0},"delivery":{"delivered":false,"surface":null,"injectionId":null,"deliveredTokens":0,"beforeFirstExecutedMistake":false},"expectedFindingIds":[],"naturalCaptureMissing":false,"graphHash":"d549ebe6b6c74b4fd1f07e3face698d5d8daa4106307d03b064823a873cbe564","repoCommit":"261c836b676de6342f5504e517a46dafd088cb70","workingTreeDirty":false,"producerPromptHash":"952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","consumerPromptHash":"572da834237a72503059843d2ef6c2b4993e01a5c1fa1901c4858a4bbc30c8b4","fixtureHash":"1dce61a64a8ce49bf92bd638c561aef84691d3d5b4fa19b0b451ab614426a968","stdoutHash":"0a9b074aa63c9d26c0e4dcd0f8a93106186e88942467fb1973259b47148c2f45","stderrHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"} +{"kind":"handoff-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","suiteVersion":1,"scenarioId":"verification-entry-point","scenarioFamily":"command-recovery-transfer","pairId":"verification-entry-point-1","repetition":0,"order":1,"arm":"natural","producer":{"client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe71e-3509-70a0-a1fe-d54b347bc1f6","correct":true,"mistakeAttempted":true,"mistakeExecuted":true,"captureSuccess":true,"capturedFindingIds":["agent-mslycwls-0-dc822bf2"],"acceptedFindingIds":["agent-mslycwls-0-dc822bf2"],"naturalCaptureDiagnostics":[{"key":"agent-mslycwls-0-dc822bf2","targetPrimaryMatch":true,"targetSecondaryMatch":true,"originAgent":true,"hasEvidence":true,"hasApplicability":true,"calibratedConfidence":true,"invalidatorCount":1,"hasScope":true,"type":"command","claimHash":"220725b6c594019cc10a9b1fec01f5b403c0756e08bddc69b5a9b2315710f71b"}],"latencyMs":77745,"exitCode":0,"timedOut":false,"uncachedInputTokens":199802,"cacheCreationInputTokens":null,"cachedInputTokens":148224,"outputTokens":2115,"totalTokens":null,"costUsd":null,"toolCalls":25,"failedToolCalls":2},"consumer":{"client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"693265da-0398-485d-8c70-a69d9faa458c","correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true,"latencyMs":37731,"exitCode":0,"timedOut":false,"uncachedInputTokens":2,"cacheCreationInputTokens":314,"cachedInputTokens":62296,"outputTokens":415,"totalTokens":63027,"costUsd":0.40236629999999995,"toolCalls":8,"failedToolCalls":0},"delivery":{"delivered":true,"surface":"session-start","injectionId":"2-6cc362d1","deliveredTokens":81,"beforeFirstExecutedMistake":true},"expectedFindingIds":["agent-mslycwls-0-dc822bf2"],"naturalCaptureMissing":false,"graphHash":"5c1948931efcb4393bd5d4b4bb6a9a8a878192b59e6b1e1b31fa12587523d7d9","repoCommit":"261c836b676de6342f5504e517a46dafd088cb70","workingTreeDirty":false,"producerPromptHash":"952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","consumerPromptHash":"572da834237a72503059843d2ef6c2b4993e01a5c1fa1901c4858a4bbc30c8b4","fixtureHash":"1dce61a64a8ce49bf92bd638c561aef84691d3d5b4fa19b0b451ab614426a968","stdoutHash":"a2a851b0e5cd4181f9de02d84c407c8d63e24682a24b97d1954082cfa72f41a2","stderrHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"} +{"kind":"handoff-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","suiteVersion":1,"scenarioId":"verification-entry-point","scenarioFamily":"command-recovery-transfer","pairId":"verification-entry-point-1","repetition":0,"order":2,"arm":"oracle","producer":{"client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe71e-3509-70a0-a1fe-d54b347bc1f6","correct":true,"mistakeAttempted":true,"mistakeExecuted":true,"captureSuccess":true,"capturedFindingIds":["agent-mslycwls-0-dc822bf2"],"acceptedFindingIds":["agent-mslycwls-0-dc822bf2"],"naturalCaptureDiagnostics":[{"key":"agent-mslycwls-0-dc822bf2","targetPrimaryMatch":true,"targetSecondaryMatch":true,"originAgent":true,"hasEvidence":true,"hasApplicability":true,"calibratedConfidence":true,"invalidatorCount":1,"hasScope":true,"type":"command","claimHash":"220725b6c594019cc10a9b1fec01f5b403c0756e08bddc69b5a9b2315710f71b"}],"latencyMs":77745,"exitCode":0,"timedOut":false,"uncachedInputTokens":199802,"cacheCreationInputTokens":null,"cachedInputTokens":148224,"outputTokens":2115,"totalTokens":null,"costUsd":null,"toolCalls":25,"failedToolCalls":2},"consumer":{"client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"37758835-0daf-4bcb-b353-30a0c07fe21d","correct":true,"firstPass":true,"mistakeAttempted":true,"mistakeExecuted":false,"proof":true,"latencyMs":91262,"exitCode":0,"timedOut":false,"uncachedInputTokens":2,"cacheCreationInputTokens":727,"cachedInputTokens":67378,"outputTokens":738,"totalTokens":68845,"costUsd":0.5952861,"toolCalls":15,"failedToolCalls":0},"delivery":{"delivered":true,"surface":"session-start","injectionId":"2-ad144b2a","deliveredTokens":82,"beforeFirstExecutedMistake":true},"expectedFindingIds":["harvested-mslyg346-0-cb391895"],"naturalCaptureMissing":false,"graphHash":"629e5a2971d5500467644f1fa23238b4d89f385a2f567fa6ecc10cab402ae033","repoCommit":"261c836b676de6342f5504e517a46dafd088cb70","workingTreeDirty":false,"producerPromptHash":"952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","consumerPromptHash":"572da834237a72503059843d2ef6c2b4993e01a5c1fa1901c4858a4bbc30c8b4","fixtureHash":"1dce61a64a8ce49bf92bd638c561aef84691d3d5b4fa19b0b451ab614426a968","stdoutHash":"d597a753220e58c0045bd078e0adca31f13a95e5ccbf2fe27789cd85b163c813","stderrHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"} +{"kind":"handoff-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","suiteVersion":1,"scenarioId":"verification-entry-point","scenarioFamily":"command-recovery-transfer","pairId":"verification-entry-point-1","repetition":0,"order":3,"arm":"irrelevant","producer":{"client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe71e-3509-70a0-a1fe-d54b347bc1f6","correct":true,"mistakeAttempted":true,"mistakeExecuted":true,"captureSuccess":true,"capturedFindingIds":["agent-mslycwls-0-dc822bf2"],"acceptedFindingIds":["agent-mslycwls-0-dc822bf2"],"naturalCaptureDiagnostics":[{"key":"agent-mslycwls-0-dc822bf2","targetPrimaryMatch":true,"targetSecondaryMatch":true,"originAgent":true,"hasEvidence":true,"hasApplicability":true,"calibratedConfidence":true,"invalidatorCount":1,"hasScope":true,"type":"command","claimHash":"220725b6c594019cc10a9b1fec01f5b403c0756e08bddc69b5a9b2315710f71b"}],"latencyMs":77745,"exitCode":0,"timedOut":false,"uncachedInputTokens":199802,"cacheCreationInputTokens":null,"cachedInputTokens":148224,"outputTokens":2115,"totalTokens":null,"costUsd":null,"toolCalls":25,"failedToolCalls":2},"consumer":{"client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"cd3512be-8aa1-4c18-82d1-39fe7ce4af67","correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true,"latencyMs":171733,"exitCode":0,"timedOut":false,"uncachedInputTokens":2,"cacheCreationInputTokens":1552,"cachedInputTokens":84302,"outputTokens":332,"totalTokens":86188,"costUsd":0.9915465000000001,"toolCalls":29,"failedToolCalls":2},"delivery":{"delivered":true,"surface":"session-start","injectionId":"2-205f4617","deliveredTokens":76,"beforeFirstExecutedMistake":true},"expectedFindingIds":["harvested-mslyi1ks-0-ccd7d06a"],"naturalCaptureMissing":false,"graphHash":"9a0be93f94cdd57dd3d4a97a0ff3e8d9b63af43df8f2e2db1e156e39ac8f390e","repoCommit":"261c836b676de6342f5504e517a46dafd088cb70","workingTreeDirty":false,"producerPromptHash":"952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","consumerPromptHash":"572da834237a72503059843d2ef6c2b4993e01a5c1fa1901c4858a4bbc30c8b4","fixtureHash":"1dce61a64a8ce49bf92bd638c561aef84691d3d5b4fa19b0b451ab614426a968","stdoutHash":"820be4cecd6fbd999dba162d4e9ff3dce5c5756512ac1f4634ea52565fb8f5dd","stderrHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"} +{"kind":"handoff-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","suiteVersion":1,"scenarioId":"verification-entry-point","scenarioFamily":"command-recovery-transfer","pairId":"verification-entry-point-1","repetition":0,"order":4,"arm":"stale","producer":{"client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe71e-3509-70a0-a1fe-d54b347bc1f6","correct":true,"mistakeAttempted":true,"mistakeExecuted":true,"captureSuccess":true,"capturedFindingIds":["agent-mslycwls-0-dc822bf2"],"acceptedFindingIds":["agent-mslycwls-0-dc822bf2"],"naturalCaptureDiagnostics":[{"key":"agent-mslycwls-0-dc822bf2","targetPrimaryMatch":true,"targetSecondaryMatch":true,"originAgent":true,"hasEvidence":true,"hasApplicability":true,"calibratedConfidence":true,"invalidatorCount":1,"hasScope":true,"type":"command","claimHash":"220725b6c594019cc10a9b1fec01f5b403c0756e08bddc69b5a9b2315710f71b"}],"latencyMs":77745,"exitCode":0,"timedOut":false,"uncachedInputTokens":199802,"cacheCreationInputTokens":null,"cachedInputTokens":148224,"outputTokens":2115,"totalTokens":null,"costUsd":null,"toolCalls":25,"failedToolCalls":2},"consumer":{"client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"e8aff96e-db2d-4475-8ef3-2cd19ba4710d","correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true,"latencyMs":66850,"exitCode":0,"timedOut":false,"uncachedInputTokens":2,"cacheCreationInputTokens":1117,"cachedInputTokens":66107,"outputTokens":105,"totalTokens":67331,"costUsd":0.5191916999999999,"toolCalls":12,"failedToolCalls":0},"delivery":{"delivered":true,"surface":"session-start","injectionId":"2-87db3a23","deliveredTokens":91,"beforeFirstExecutedMistake":true},"expectedFindingIds":["harvested-mslylq5m-0-735a3c17"],"naturalCaptureMissing":false,"graphHash":"f91f626d8f521003b96ca482105c768f52b3fa5bad558cd9c90efb12934ab947","repoCommit":"261c836b676de6342f5504e517a46dafd088cb70","workingTreeDirty":false,"producerPromptHash":"952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","consumerPromptHash":"572da834237a72503059843d2ef6c2b4993e01a5c1fa1901c4858a4bbc30c8b4","fixtureHash":"1dce61a64a8ce49bf92bd638c561aef84691d3d5b4fa19b0b451ab614426a968","stdoutHash":"c16c0e2221957e07f23dfab2e37c945bc066aede9e81f1c917e6379ef96cea0e","stderrHash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"} diff --git a/evals/results/2026-08-09-pr302-codex-to-codex-natural.jsonl b/evals/results/2026-08-09-pr302-codex-to-codex-natural.jsonl new file mode 100644 index 000000000..db854765d --- /dev/null +++ b/evals/results/2026-08-09-pr302-codex-to-codex-natural.jsonl @@ -0,0 +1 @@ +{"kind":"handoff-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","suiteVersion":1,"scenarioId":"verification-entry-point","scenarioFamily":"command-recovery-transfer","pairId":"verification-entry-point-1","repetition":0,"order":0,"arm":"natural","producer":{"client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe72f-2826-7d01-9d86-030365a32080","correct":true,"mistakeAttempted":true,"mistakeExecuted":true,"captureSuccess":true,"capturedFindingIds":["agent-mslz0oy2-0-d5fe7cdb"],"acceptedFindingIds":["agent-mslz0oy2-0-d5fe7cdb"],"naturalCaptureDiagnostics":[{"key":"agent-mslz0oy2-0-d5fe7cdb","targetPrimaryMatch":true,"targetSecondaryMatch":true,"originAgent":true,"hasEvidence":true,"hasApplicability":true,"calibratedConfidence":true,"invalidatorCount":1,"hasScope":true,"type":"command","claimHash":"ff0738b345ee659e89a520eb16dcf40a54e8f16fe1014a811dd3730bc2504d1f"}],"latencyMs":77324,"exitCode":0,"timedOut":false,"uncachedInputTokens":231566,"cacheCreationInputTokens":null,"cachedInputTokens":211456,"outputTokens":1961,"totalTokens":null,"costUsd":null,"toolCalls":27,"failedToolCalls":1},"consumer":{"client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe730-56a2-7a11-bc88-1bae67bfd539","correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true,"latencyMs":91419,"exitCode":0,"timedOut":false,"uncachedInputTokens":265079,"cacheCreationInputTokens":null,"cachedInputTokens":197632,"outputTokens":2087,"totalTokens":null,"costUsd":null,"toolCalls":37,"failedToolCalls":0},"delivery":{"delivered":true,"surface":"session-start","injectionId":"2-bd366a37","deliveredTokens":81,"beforeFirstExecutedMistake":true},"expectedFindingIds":["agent-mslz0oy2-0-d5fe7cdb"],"naturalCaptureMissing":false,"graphHash":"66ae7029d13d12544b86747421fd868eb5b0eb3cc498f6045c7a43f5528663e4","repoCommit":"261c836b676de6342f5504e517a46dafd088cb70","workingTreeDirty":false,"producerPromptHash":"952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","consumerPromptHash":"572da834237a72503059843d2ef6c2b4993e01a5c1fa1901c4858a4bbc30c8b4","fixtureHash":"1dce61a64a8ce49bf92bd638c561aef84691d3d5b4fa19b0b451ab614426a968","stdoutHash":"9462343784f0fe383f28f5139a29d20961f5c922e45636081f5177cc315bb867","stderrHash":"1aa26269eb1cc57f86b235a03cda53c004edb5b1e9fc99d4da4f00843293d721"} diff --git a/evals/results/2026-08-09-pr302-concurrent-codex-to-claude.jsonl b/evals/results/2026-08-09-pr302-concurrent-codex-to-claude.jsonl new file mode 100644 index 000000000..83026e265 --- /dev/null +++ b/evals/results/2026-08-09-pr302-concurrent-codex-to-claude.jsonl @@ -0,0 +1,2 @@ +{"kind":"concurrency-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","pairId":"concurrent-1","repetition":0,"order":0,"arm":"empty","writerCount":3,"producerClient":"codex","producerModel":"gpt-5.6-sol","writers":[{"scenarioId":"verification-entry-point","episodeId":"concurrent-1-writer-verification-entry-point-codex","client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true,"captureSuccess":true,"capturedFindingIds":["agent-mslzlpab-0-92ce1645"],"acceptedFindingIds":["agent-mslzlpab-0-92ce1645"],"exitCode":0,"timedOut":false,"latencyMs":102223,"uncachedInputTokens":308250,"cacheCreationInputTokens":null,"cachedInputTokens":275200,"outputTokens":2666,"totalTokens":null,"costUsd":null,"toolCalls":35,"failedToolCalls":3},{"scenarioId":"generated-source-of-truth","episodeId":"concurrent-1-writer-generated-source-of-truth-codex","client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true,"captureSuccess":true,"capturedFindingIds":["agent-mslzlk4x-0-f9de90e4"],"acceptedFindingIds":["agent-mslzlk4x-0-f9de90e4"],"exitCode":0,"timedOut":false,"latencyMs":99335,"uncachedInputTokens":243170,"cacheCreationInputTokens":null,"cachedInputTokens":213248,"outputTokens":2694,"totalTokens":null,"costUsd":null,"toolCalls":31,"failedToolCalls":1},{"scenarioId":"false-positive-validation","episodeId":"concurrent-1-writer-false-positive-validation-codex","client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true,"captureSuccess":true,"capturedFindingIds":["agent-mslzme9b-0-73b27384"],"acceptedFindingIds":["agent-mslzme9b-0-73b27384"],"exitCode":0,"timedOut":false,"latencyMs":138956,"uncachedInputTokens":357359,"cacheCreationInputTokens":null,"cachedInputTokens":289536,"outputTokens":3258,"totalTokens":null,"costUsd":null,"toolCalls":53,"failedToolCalls":2}],"captureSuccesses":3,"acceptedFindingIds":["agent-mslzlpab-0-92ce1645","agent-mslzlk4x-0-f9de90e4","agent-mslzme9b-0-73b27384"],"integrity":{"graphLines":99,"snapshotLines":24,"malformedLines":0,"orphanedFindings":0,"contradictedFindings":0,"activeFindings":3,"acceptedWritesObserved":3,"acceptanceObservable":true,"missingAcceptedWrites":[],"zeroLoss":true,"parseable":true},"consumer":{"client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"4928ef5e-4d30-49e9-bfd0-f9f7337a772b","correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"scenarioResults":{"verification-entry-point":{"correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true},"generated-source-of-truth":{"correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true},"false-positive-validation":{"correct":true,"firstPass":true,"mistakeAttempted":true,"mistakeExecuted":false,"proof":true}},"exitCode":0,"timedOut":false,"latencyMs":190656,"uncachedInputTokens":2,"cacheCreationInputTokens":1019,"cachedInputTokens":78768,"outputTokens":719,"totalTokens":80508,"costUsd":1.0522956,"toolCalls":34,"failedToolCalls":1},"delivery":{"expected":0,"delivered":0,"allBeforeFirstAction":null,"deliveredTokens":0},"graphHash":"c3915bf6712022ab77ffd7ed22f6868e248fdb27e71e397030a2079ae83d8f22","producerPromptHashes":["952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","281d98b09fe2509c33892e36dfb172ae98de5eb0d869dc27744f260dba9d81b6","e2b7b6b21f86b8d45972ecc27b4c6240fa47478eec212dec797562d15ec696b0"],"consumerPromptHash":"41ea1dc2a3f7723dc2747c2b192ba3daaafc44cb69da8373f5e674e0d3f72223","repoCommit":"712b0d3c4541c91a710ca4921e8db8d2b665ddb3","workingTreeDirty":false} +{"kind":"concurrency-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","pairId":"concurrent-1","repetition":0,"order":1,"arm":"natural","writerCount":3,"producerClient":"codex","producerModel":"gpt-5.6-sol","writers":[{"scenarioId":"verification-entry-point","episodeId":"concurrent-1-writer-verification-entry-point-codex","client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true,"captureSuccess":true,"capturedFindingIds":["agent-mslzlpab-0-92ce1645"],"acceptedFindingIds":["agent-mslzlpab-0-92ce1645"],"exitCode":0,"timedOut":false,"latencyMs":102223,"uncachedInputTokens":308250,"cacheCreationInputTokens":null,"cachedInputTokens":275200,"outputTokens":2666,"totalTokens":null,"costUsd":null,"toolCalls":35,"failedToolCalls":3},{"scenarioId":"generated-source-of-truth","episodeId":"concurrent-1-writer-generated-source-of-truth-codex","client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true,"captureSuccess":true,"capturedFindingIds":["agent-mslzlk4x-0-f9de90e4"],"acceptedFindingIds":["agent-mslzlk4x-0-f9de90e4"],"exitCode":0,"timedOut":false,"latencyMs":99335,"uncachedInputTokens":243170,"cacheCreationInputTokens":null,"cachedInputTokens":213248,"outputTokens":2694,"totalTokens":null,"costUsd":null,"toolCalls":31,"failedToolCalls":1},{"scenarioId":"false-positive-validation","episodeId":"concurrent-1-writer-false-positive-validation-codex","client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true,"captureSuccess":true,"capturedFindingIds":["agent-mslzme9b-0-73b27384"],"acceptedFindingIds":["agent-mslzme9b-0-73b27384"],"exitCode":0,"timedOut":false,"latencyMs":138956,"uncachedInputTokens":357359,"cacheCreationInputTokens":null,"cachedInputTokens":289536,"outputTokens":3258,"totalTokens":null,"costUsd":null,"toolCalls":53,"failedToolCalls":2}],"captureSuccesses":3,"acceptedFindingIds":["agent-mslzlpab-0-92ce1645","agent-mslzlk4x-0-f9de90e4","agent-mslzme9b-0-73b27384"],"integrity":{"graphLines":99,"snapshotLines":24,"malformedLines":0,"orphanedFindings":0,"contradictedFindings":0,"activeFindings":3,"acceptedWritesObserved":3,"acceptanceObservable":true,"missingAcceptedWrites":[],"zeroLoss":true,"parseable":true},"consumer":{"client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","sessionId":"fa24d375-d10c-4f4b-8161-f18756be2400","correct":false,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"scenarioResults":{"verification-entry-point":{"correct":false,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":false,"proof":false},"generated-source-of-truth":{"correct":false,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":false},"false-positive-validation":{"correct":false,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":false,"proof":false}},"exitCode":0,"timedOut":false,"latencyMs":167666,"uncachedInputTokens":2,"cacheCreationInputTokens":1213,"cachedInputTokens":76785,"outputTokens":584,"totalTokens":78584,"costUsd":0.9835350000000003,"toolCalls":23,"failedToolCalls":1},"delivery":{"expected":3,"delivered":3,"allBeforeFirstAction":true,"deliveredTokens":141},"graphHash":"c3915bf6712022ab77ffd7ed22f6868e248fdb27e71e397030a2079ae83d8f22","producerPromptHashes":["952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","281d98b09fe2509c33892e36dfb172ae98de5eb0d869dc27744f260dba9d81b6","e2b7b6b21f86b8d45972ecc27b4c6240fa47478eec212dec797562d15ec696b0"],"consumerPromptHash":"41ea1dc2a3f7723dc2747c2b192ba3daaafc44cb69da8373f5e674e0d3f72223","repoCommit":"712b0d3c4541c91a710ca4921e8db8d2b665ddb3","workingTreeDirty":false} diff --git a/evals/results/2026-08-09-pr302-concurrent-diagnostic-invalid-attribution.jsonl b/evals/results/2026-08-09-pr302-concurrent-diagnostic-invalid-attribution.jsonl new file mode 100644 index 000000000..6cb050c1e --- /dev/null +++ b/evals/results/2026-08-09-pr302-concurrent-diagnostic-invalid-attribution.jsonl @@ -0,0 +1,2 @@ +{"kind":"concurrency-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","pairId":"concurrent-1","repetition":0,"order":0,"arm":"empty","writerCount":3,"producerClient":"claude","producerModel":"claude-sonnet-5","writers":[{"scenarioId":"verification-entry-point","episodeId":"concurrent-1-writer-verification-entry-point-claude","client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true,"captureSuccess":true,"capturedFindingIds":["agent-mslz9qol-0-d5130689"],"acceptedFindingIds":["agent-mslz9qol-0-d5130689"],"exitCode":0,"timedOut":false,"latencyMs":88200,"uncachedInputTokens":2,"cacheCreationInputTokens":1410,"cachedInputTokens":69226,"outputTokens":266,"totalTokens":70904,"costUsd":0.6040107,"toolCalls":15,"failedToolCalls":0},{"scenarioId":"generated-source-of-truth","episodeId":"concurrent-1-writer-generated-source-of-truth-claude","client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true,"captureSuccess":true,"capturedFindingIds":["agent-mslz9aha-0-4f6c71f0"],"acceptedFindingIds":["agent-mslz9aha-0-4f6c71f0"],"exitCode":0,"timedOut":false,"latencyMs":66109,"uncachedInputTokens":2,"cacheCreationInputTokens":1056,"cachedInputTokens":66072,"outputTokens":108,"totalTokens":67238,"costUsd":0.5743772999999999,"toolCalls":17,"failedToolCalls":1},{"scenarioId":"false-positive-validation","episodeId":"concurrent-1-writer-false-positive-validation-claude","client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true,"captureSuccess":true,"capturedFindingIds":["agent-mslz9qol-0-d5130689"],"acceptedFindingIds":[],"exitCode":0,"timedOut":false,"latencyMs":171465,"uncachedInputTokens":2,"cacheCreationInputTokens":3108,"cachedInputTokens":73214,"outputTokens":1343,"totalTokens":77667,"costUsd":0.91197,"toolCalls":20,"failedToolCalls":0}],"captureSuccesses":3,"acceptedFindingIds":["agent-mslz9qol-0-d5130689","agent-mslz9aha-0-4f6c71f0"],"integrity":{"graphLines":268,"snapshotLines":59,"malformedLines":0,"orphanedFindings":0,"contradictedFindings":0,"activeFindings":2,"acceptedWritesObserved":2,"acceptanceObservable":true,"missingAcceptedWrites":[],"zeroLoss":true,"parseable":true},"consumer":{"client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe737-fb8a-7813-9e8b-3457fbc36e20","correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"scenarioResults":{"verification-entry-point":{"correct":true,"firstPass":true,"mistakeAttempted":true,"mistakeExecuted":false,"proof":true},"generated-source-of-truth":{"correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true},"false-positive-validation":{"correct":true,"firstPass":true,"mistakeAttempted":true,"mistakeExecuted":false,"proof":true}},"exitCode":0,"timedOut":false,"latencyMs":163298,"uncachedInputTokens":464393,"cacheCreationInputTokens":null,"cachedInputTokens":419072,"outputTokens":4931,"totalTokens":null,"costUsd":null,"toolCalls":50,"failedToolCalls":1},"delivery":{"expected":0,"delivered":0,"allBeforeFirstAction":null,"deliveredTokens":0},"graphHash":"0bcf201305bba7b876f971793e19e8af0baa0584a6cbd5b25e882b612ee0fee8","producerPromptHashes":["952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","281d98b09fe2509c33892e36dfb172ae98de5eb0d869dc27744f260dba9d81b6","e2b7b6b21f86b8d45972ecc27b4c6240fa47478eec212dec797562d15ec696b0"],"consumerPromptHash":"41ea1dc2a3f7723dc2747c2b192ba3daaafc44cb69da8373f5e674e0d3f72223","repoCommit":"1477745dbdf90314e2c05a7ce12f03c23169aff6","workingTreeDirty":false} +{"kind":"concurrency-run","schemaVersion":3,"suite":"token-optimizer-cross-client-mistake-transfer","pairId":"concurrent-1","repetition":0,"order":1,"arm":"natural","writerCount":3,"producerClient":"claude","producerModel":"claude-sonnet-5","writers":[{"scenarioId":"verification-entry-point","episodeId":"concurrent-1-writer-verification-entry-point-claude","client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true,"captureSuccess":true,"capturedFindingIds":["agent-mslz9qol-0-d5130689"],"acceptedFindingIds":["agent-mslz9qol-0-d5130689"],"exitCode":0,"timedOut":false,"latencyMs":88200,"uncachedInputTokens":2,"cacheCreationInputTokens":1410,"cachedInputTokens":69226,"outputTokens":266,"totalTokens":70904,"costUsd":0.6040107,"toolCalls":15,"failedToolCalls":0},{"scenarioId":"generated-source-of-truth","episodeId":"concurrent-1-writer-generated-source-of-truth-claude","client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true,"captureSuccess":true,"capturedFindingIds":["agent-mslz9aha-0-4f6c71f0"],"acceptedFindingIds":["agent-mslz9aha-0-4f6c71f0"],"exitCode":0,"timedOut":false,"latencyMs":66109,"uncachedInputTokens":2,"cacheCreationInputTokens":1056,"cachedInputTokens":66072,"outputTokens":108,"totalTokens":67238,"costUsd":0.5743772999999999,"toolCalls":17,"failedToolCalls":1},{"scenarioId":"false-positive-validation","episodeId":"concurrent-1-writer-false-positive-validation-claude","client":"claude","clientVersion":"2.1.225","model":"claude-sonnet-5","modelVersion":"claude-sonnet-5","correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true,"captureSuccess":true,"capturedFindingIds":["agent-mslz9qol-0-d5130689"],"acceptedFindingIds":[],"exitCode":0,"timedOut":false,"latencyMs":171465,"uncachedInputTokens":2,"cacheCreationInputTokens":3108,"cachedInputTokens":73214,"outputTokens":1343,"totalTokens":77667,"costUsd":0.91197,"toolCalls":20,"failedToolCalls":0}],"captureSuccesses":3,"acceptedFindingIds":["agent-mslz9qol-0-d5130689","agent-mslz9aha-0-4f6c71f0"],"integrity":{"graphLines":268,"snapshotLines":59,"malformedLines":0,"orphanedFindings":0,"contradictedFindings":0,"activeFindings":2,"acceptedWritesObserved":2,"acceptanceObservable":true,"missingAcceptedWrites":[],"zeroLoss":true,"parseable":true},"consumer":{"client":"codex","clientVersion":"0.147.0","model":"gpt-5.6-sol","modelVersion":null,"sessionId":"019fe73a-79c9-7790-8f2a-2661092d1855","correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"scenarioResults":{"verification-entry-point":{"correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true},"generated-source-of-truth":{"correct":true,"firstPass":false,"mistakeAttempted":true,"mistakeExecuted":true,"proof":true},"false-positive-validation":{"correct":true,"firstPass":true,"mistakeAttempted":false,"mistakeExecuted":false,"proof":true}},"exitCode":0,"timedOut":false,"latencyMs":149170,"uncachedInputTokens":514215,"cacheCreationInputTokens":null,"cachedInputTokens":432384,"outputTokens":4796,"totalTokens":null,"costUsd":null,"toolCalls":87,"failedToolCalls":1},"delivery":{"expected":2,"delivered":2,"allBeforeFirstAction":true,"deliveredTokens":111},"graphHash":"0bcf201305bba7b876f971793e19e8af0baa0584a6cbd5b25e882b612ee0fee8","producerPromptHashes":["952adf8543b15ae11b26c4b8479eefc521631ad1f2aa4be5b46900c124a37ad6","281d98b09fe2509c33892e36dfb172ae98de5eb0d869dc27744f260dba9d81b6","e2b7b6b21f86b8d45972ecc27b4c6240fa47478eec212dec797562d15ec696b0"],"consumerPromptHash":"41ea1dc2a3f7723dc2747c2b192ba3daaafc44cb69da8373f5e674e0d3f72223","repoCommit":"1477745dbdf90314e2c05a7ce12f03c23169aff6","workingTreeDirty":false} diff --git a/evals/results/2026-08-09-pr302-cross-client-mistake-transfer.md b/evals/results/2026-08-09-pr302-cross-client-mistake-transfer.md new file mode 100644 index 000000000..5fd0020f1 --- /dev/null +++ b/evals/results/2026-08-09-pr302-cross-client-mistake-transfer.md @@ -0,0 +1,266 @@ +# PR #302 cross-client mistake-transfer cohort — August 9, 2026 + +## Verdict + +The live graph's storage and cross-client delivery chain works in the tested +cohort. Its effectiveness is **not established**, and one concurrent cohort was +harmful. + +- Codex `0.147.0` / `gpt-5.6-sol` and Claude Code `2.1.225` / + `claude-sonnet-5` each naturally harvested a qualifying finding. +- Fresh same-client and cross-client consumers received the exact producer + finding at `SessionStart`, before the first preventable action. +- The matched cross-client controls had 10/10 correct consumers and 0/10 + executed recurrences. Because both empty arms already had zero recurrence, + the cohort cannot show a recurrence reduction. +- Three concurrent Codex writers produced three independently acknowledged + findings. The graph had zero malformed lines, orphaned findings, or lost + accepted writes, and a later Claude consumer received 3/3 findings. +- That natural concurrent consumer failed all three hidden end-state/proof + graders while its empty-graph match passed. Correctness takes precedence over + the natural arm's lower token, tool, latency, and cost totals. +- Irrelevant and stale controls were listed at `SessionStart`. They caused no + correctness regression in the single-task cross-client cohort, but irrelevant + delivery failed the stricter negative-control requirement and was expensive. +- The graph text was only 81 tokens in each single-finding handoff and 141 tokens + in the three-finding concurrent handoff. A no-system comparison nevertheless + found substantial full-integration overhead, especially for Codex. The likely + issue is the plugin/MCP tool and behavior surface, not graph text alone. + +The dashboard therefore reports `insufficient randomized evidence`; no +"smarter" or net-efficiency claim is made. + +## Protocol and isolation + +- Pre-registration: `docs/CROSS_CLIENT_MISTAKE_TRANSFER_PROTOCOL.md`. +- Scenario reported here: `verification-entry-point` for the five-arm and + same-client cohorts; all three registered scenarios for concurrency. +- Consumer arms: `empty`, `natural`, `oracle`, `irrelevant`, and `stale`. +- Every matched arm received the same frozen post-producer workspace. CLI + transcripts, session state, `AGENTS.md`, `CLAUDE.md`, auto-memory, `.codex`, + `.claude`, and `.token-optimizer` were not copied. +- Codex used a PR-only local plugin cache and a config-level MCP command pinned + to this checkout's built `dist/server/index.js`. The MCP graph, shared graph, + state, audit, episode, pair, task, model, and client values were passed + explicitly because Codex does not inherit those environment variables into an + MCP subprocess. +- Claude used only project settings, an explicit PR plugin directory, a strict + local MCP config, and non-persistent print sessions. +- The no-system Codex baseline used `--ignore-user-config`, zero project-document + budget, memory off, and no plugins/MCP. The Claude baseline used no plugin and + a strict empty MCP config. +- Hidden audit sentinels and filesystem end state determined correctness. Model + prose was never the grader. +- Published ledgers contain hashes, identities, finding IDs, usage, timing, + grades, and redacted diagnostics; no transcripts are included. + +## Matched cross-client controls + +### Codex producer → Claude consumer + +All rows used clean commit `261c836` and one matched pair. + +| Arm | Correct | Attempted | Executed | Delivery | Injected | Total tokens | Tools | Latency | Cost | +| --- | :---: | :---: | :---: | --- | ---: | ---: | ---: | ---: | ---: | +| Empty | yes | no | no | — | 0 | 72,326 | 19 | 102.933 s | $0.651624 | +| Natural | yes | no | no | SessionStart | 81 | 63,027 | 8 | 37.731 s | $0.402366 | +| Oracle | yes | yes | no | SessionStart | 82 | 68,845 | 15 | 91.262 s | $0.595286 | +| Irrelevant | yes | no | no | SessionStart | 76 | 86,188 | 29 | 171.733 s | $0.991547 | +| Stale | yes | no | no | SessionStart | 91 | 67,331 | 12 | 66.850 s | $0.519192 | + +Natural versus empty saved 9,299 total tokens, 11 tool calls, 65.202 seconds, +and $0.249257 in this pair. The paired recurrence effect was exactly zero +because neither arm executed the mistake. The irrelevant arm was the slowest, +most expensive, and most tool-heavy arm. + +### Claude producer → Codex consumer + +All rows used clean commit `261c836` and one matched pair. Codex did not expose a +single total-token or price value, so uncached/cached accounting is reported +without manufacturing a total or cost. + +| Arm | Correct | Attempted | Executed | Delivery | Injected | Uncached | Cached | Tools | Latency | +| --- | :---: | :---: | :---: | --- | ---: | ---: | ---: | ---: | ---: | +| Empty | yes | no | no | — | 0 | 374,157 | 335,616 | 49 | 94.234 s | +| Natural | yes | no | no | SessionStart | 81 | 229,702 | 167,168 | 31 | 74.203 s | +| Oracle | yes | no | no | SessionStart | 82 | 223,012 | 189,440 | 23 | 62.278 s | +| Irrelevant | yes | no | no | SessionStart | 76 | 288,577 | 264,448 | 52 | 112.690 s | +| Stale | yes | no | no | SessionStart | 91 | 183,261 | 166,656 | 18 | 78.059 s | + +Natural versus empty used 144,455 fewer uncached tokens, 168,448 fewer cached +tokens, 18 fewer tools, and 20.031 fewer seconds in this pair. Again, recurrence +was already zero in empty, so prevention effectiveness is not identified. + +## Same-client fresh sessions + +| Producer → consumer | Capture | Exact delivery | Attempted | Executed | Correct | Injected | Consumer usage | +| --- | :---: | :---: | :---: | :---: | :---: | ---: | --- | +| Codex → Codex | yes | yes | no | no | yes | 81 | 265,079 uncached; 37 tools; 91.419 s | +| Claude → Claude | yes | yes | no | no | yes | 81 | 67,245 total; 14 tools; 76.906 s; $0.589651 | + +These rows prove a graph-only handoff to a fresh process/session. With no matched +empty arms, they are not causal effectiveness estimates. + +## Concurrent writers and later consumer + +The accepted corrected cohort used clean commit `712b0d3`. + +### Writer integrity + +- Three Codex processes ran concurrently against one workspace and graph. +- All three completed correctly after the required producer mistake. +- Natural semantic capture: 3/3 writers. +- Acknowledged unique findings: 3. +- Graph lines: 99; snapshot lines: 24. +- Malformed lines: 0; orphaned findings: 0; contradicted findings: 0. +- Missing accepted writes: 0; graph parseable: yes; zero-loss gate: yes. +- Later natural Claude consumer received 3/3 exact finding IDs at SessionStart + before its first task action, at a total of 141 injected tokens. + +### Consumer outcome + +| Arm | Correct | First pass | Attempted | Executed | Total tokens | Tools | Latency | Cost | +| --- | :---: | :---: | :---: | :---: | ---: | ---: | ---: | ---: | +| Empty | yes | no | yes | yes | 80,508 | 34 | 190.656 s | $1.052296 | +| Natural | **no** | no | yes | yes | 78,584 | 23 | 167.666 s | $0.983535 | + +Natural saved 1,924 tokens, 11 tools, 22.990 seconds, and $0.068761, but its +correctness delta was -100 percentage points. It failed the required proof/end +state for verification, generated-source, and false-positive-validation. This +row is harmful by the pre-registered priority ordering. + +### Retained diagnostic cohort + +An earlier Claude×3 → Codex diagnostic exposed a grading defect: a writer with +no acknowledged `wiki_write` was incorrectly credited when another writer's +finding in the shared graph passed its semantic matcher. The raw ledger is +retained with `invalid-attribution` in its filename. Its aggregate +`captureSuccesses: 3` field is invalid; recomputing from per-writer acknowledged +IDs gives 2/3 writers, two accepted writes, zero lost writes, and 2/2 later +delivery. The evaluator now requires both semantic match and that writer's +acknowledged finding ID; a regression test covers the case. + +The diagnostic consumers were both correct, but both repeated the generated +source-of-truth mistake. It is not used in the dashboard claim cohort. + +## Context-window and integration overhead + +The same hidden-grader task was also run without the plugin or any MCP server. +These are single-run diagnostics, not randomized estimates. + +### Codex consumer + +| Configuration | Correct | Attempted | Executed | Uncached | Cached | Tools | Latency | +| --- | :---: | :---: | :---: | ---: | ---: | ---: | ---: | +| No system | yes | yes | no | 83,464 | 72,192 | 10 | 33.180 s | +| Full integration, empty graph | yes | no | no | 374,157 | 335,616 | 49 | 94.234 s | +| Full integration, natural graph | yes | no | no | 229,702 | 167,168 | 31 | 74.203 s | + +Against no-system, the empty integration added 290,693 uncached tokens +(+348.3%), 39 tools, and 61.054 seconds. Natural retrieval reduced the empty +integration's work but remained 146,238 uncached tokens (+175.2%), 21 tools, and +41.023 seconds above no-system. The 81-token graph injection was only 0.035% of +the natural run's uncached accounting. + +### Claude consumer + +| Configuration | Correct | Attempted | Executed | Total tokens | Tools | Latency | Cost | +| --- | :---: | :---: | :---: | ---: | ---: | ---: | ---: | +| No system | yes | no | no | 64,332 | 14 | 47.095 s | $0.387555 | +| Full integration, empty graph | yes | no | no | 72,326 | 19 | 102.933 s | $0.651624 | +| Full integration, natural graph | yes | no | no | 63,027 | 8 | 37.731 s | $0.402366 | + +Against no-system, the empty integration added 7,994 tokens (+12.4%), five +tools, 55.838 seconds, and $0.264069. Natural used 1,305 fewer tokens, six fewer +tools, and 9.364 fewer seconds than no-system, but cost $0.014811 more. The +81-token graph injection was 0.129% of natural total tokens. + +These measurements reject the broad theory that graph text itself consumes most +of the context window. They support a narrower concern: the complete plugin/MCP +surface can induce substantial extra model and tool activity, especially in +Codex. Tool-schema prompt cost is not separately observable in these CLI +streams, so it remains part of the full-integration delta rather than a claimed +root cause. + +## Pre-registered gates + +| Gate | Result | +| --- | --- | +| 10 matched pairs per cross-client direction | **fail** — 1 per direction | +| Natural capture at least 80% | pass in accepted single/concurrent cohorts | +| Executed recurrence reduction at least 50% | **not identified** — empty recurrence was 0% | +| Paired 95% interval excludes zero | **fail** — one pair and zero effect | +| Natural correctness no more than 10 pp below empty | pass in single-task cohorts; **fail** in concurrent cohort (-100 pp) | +| Pre-action delivery at least 80% | pass — 100% of expected accepted findings | +| Irrelevant/stale controls cause no correctness regression | pass at n=1 | +| Irrelevant guidance is not delivered | **fail** — listed at SessionStart | +| Concurrent graph parseable, no orphan/lost accepted writes | pass | +| Concurrent later consumer correct with all findings | **fail** in accepted Codex×3 → Claude cohort | + +Overall status: **infrastructure demonstrated; effectiveness not established; +one harmful cohort observed.** + +## Live dashboard proof + +The eight accepted ledgers plus the corrected concurrency cohort were loaded +into one graph and served by the built dashboard at `http://localhost:3342`. + +- Evidence API: 14 handoff rows, two concurrency rows, six transfer cohorts. +- Browser: HTTP 200; six transfer rows rendered; concurrent writer table + rendered; no console or page errors. +- Dashboard verdict: `insufficient randomized evidence`. +- Concurrent row: 100% capture, integrity, and delivery; 0% natural consumer + correctness. +- Existing deterministic UI verification also passed 28/28 checks. + +The live aggregate and screenshot are local runtime artifacts; the immutable +input ledgers below are committed. + +## Redacted ledgers and SHA-256 + +| Ledger | SHA-256 | +| --- | --- | +| `2026-08-09-pr302-codex-to-claude-controls.jsonl` | `967AB5BF6AD1996BC3BD50AC8D67C675EBD3B95491EC7F7FBF5966544D93E092` | +| `2026-08-09-pr302-claude-to-codex-controls.jsonl` | `95CFE77ED878122BA028BC315A3154812365CE2B8DE8E7E5A9547B83A49B1428` | +| `2026-08-09-pr302-codex-to-codex-natural.jsonl` | `488DC043FF14F4CF314078DB157A9D1225B796FB2EDFE384FA3938522DCFD8D4` | +| `2026-08-09-pr302-claude-to-claude-natural.jsonl` | `EF552E18ECDA79729D6253042D29CDCCE26A3891814D396C302E6CD58BE1FF76` | +| `2026-08-09-pr302-concurrent-codex-to-claude.jsonl` | `477FF6F1A6D2229F6BB82F5FDC4E16316F11335FBCB71284F2DC666AF775C3FD` | +| `2026-08-09-pr302-concurrent-diagnostic-invalid-attribution.jsonl` | `EFDCCED7886E98DF85F933AD173106E2C212486CCE2682B8127C0ECC2DBD9CF2` | +| `2026-08-09-pr302-codex-no-system-baseline.jsonl` | `8DBF68AE4C2D497C097247C71C07AC0DE3E4FE0183FA19C3E312D2B45438816D` | +| `2026-08-09-pr302-claude-no-system-baseline.jsonl` | `14A7CD851249221A1DF695B7AA81068F497C6F62F721B07B3E502AE268767309` | + +## Validation + +- Repository Jest suite excluding stale `worktrees/`: 169 suites passed; + 2,126 tests passed, five skipped. +- Focused handoff/causal tests after evaluator hardening: 15/15 passed. +- Build and dashboard build passed. +- Dashboard browser verification: 28/28 checks passed. +- Client configuration verification: 227/227 checks passed. +- Client certification: 16/16 checks passed. +- Hook synchronization, package validation, and formatting checks passed. +- Lint: zero errors (492 pre-existing warnings). +- GitHub checks at evidence time: all required checks passed on the pre-report + head; the report commit triggers the final check set. + +An unscoped `npm test -- --runInBand` also discovered a stale checkout under +`worktrees/us-nf-semantic-caching` and ran its obsolete tests. Three failures +there are not from the current checkout; the current repository's corresponding +semantic-caching tests pass. The scoped current-checkout suite above is the +reported result. + +## Required next experiments before an effectiveness claim + +1. Replace unconditional SessionStart listing with prompt-aware relevance for + clients that expose a prompt lifecycle, and suppress irrelevant/stale items + where relevance cannot be established. +2. Diagnose the harmful three-finding Claude consumer with a preserved, + privacy-reviewed trace and add a regression scenario before expanding the + sample. +3. Measure MCP tool-schema and skill-loading cost separately from tool-use + behavior; reduce the default tool surface and avoid unnecessary skill reads. +4. Make the single-task fixtures harder enough that empty arms have non-zero + recurrence without telling consumers the answer, then run the pre-registered + 10 matched pairs per direction. +5. Repeat concurrency in both directions after the relevance fix and require + every writer to acknowledge its own target finding. From 24905dfc5467685500f11213066270adfa275f9f Mon Sep 17 00:00:00 2001 From: Franklin Moormann Date: Sun, 9 Aug 2026 14:07:13 -0400 Subject: [PATCH 7/9] fix: make cross-client evidence gates fail closed --- .../mistake-transfer/scripts/audit.mjs | 16 ++++- hooks-core/adapter.mjs | 2 +- hooks-core/inject.mjs | 31 ++++++---- hooks-core/metrics.mjs | 49 ++++++++++----- .../hooks/token-optimizer/lib/adapter.mjs | 2 +- .../hooks/token-optimizer/lib/inject.mjs | 31 ++++++---- .../hooks/token-optimizer/lib/metrics.mjs | 49 ++++++++++----- integrations/codex/hooks/lib/adapter.mjs | 2 +- integrations/codex/hooks/lib/inject.mjs | 31 ++++++---- integrations/codex/hooks/lib/metrics.mjs | 49 ++++++++++----- .../codex/plugin/hooks/lib/adapter.mjs | 2 +- .../codex/plugin/hooks/lib/inject.mjs | 31 ++++++---- .../codex/plugin/hooks/lib/metrics.mjs | 49 ++++++++++----- .../copilot/.github/hooks/lib/adapter.mjs | 2 +- .../copilot/.github/hooks/lib/inject.mjs | 31 ++++++---- .../copilot/.github/hooks/lib/metrics.mjs | 49 ++++++++++----- integrations/cursor/hooks/lib/adapter.mjs | 2 +- integrations/cursor/hooks/lib/inject.mjs | 31 ++++++---- integrations/cursor/hooks/lib/metrics.mjs | 49 ++++++++++----- integrations/gemini/hooks/lib/adapter.mjs | 2 +- integrations/gemini/hooks/lib/inject.mjs | 31 ++++++---- integrations/gemini/hooks/lib/metrics.mjs | 49 ++++++++++----- integrations/kilo/hooks/lib/adapter.mjs | 2 +- integrations/kilo/hooks/lib/inject.mjs | 31 ++++++---- integrations/kilo/hooks/lib/metrics.mjs | 49 ++++++++++----- integrations/opencode/hooks/lib/adapter.mjs | 2 +- integrations/opencode/hooks/lib/inject.mjs | 31 ++++++---- integrations/opencode/hooks/lib/metrics.mjs | 49 ++++++++++----- integrations/qwen/hooks/lib/adapter.mjs | 2 +- integrations/qwen/hooks/lib/inject.mjs | 31 ++++++---- integrations/qwen/hooks/lib/metrics.mjs | 49 ++++++++++----- integrations/windsurf/hooks/lib/adapter.mjs | 2 +- integrations/windsurf/hooks/lib/inject.mjs | 31 ++++++---- integrations/windsurf/hooks/lib/metrics.mjs | 49 ++++++++++----- plugin/hooks/lib/adapter.mjs | 2 +- plugin/hooks/lib/inject.mjs | 31 ++++++---- plugin/hooks/lib/metrics.mjs | 49 ++++++++++----- scripts/run-concurrent-handoff-eval.mjs | 59 ++++++++++++------- scripts/run-evidence-eval.mjs | 23 +++++--- scripts/run-handoff-eval.mjs | 47 +++++++++++---- scripts/verify-wiki-ui.mjs | 7 ++- tests/hooks/causal-evidence.test.mjs | 40 ++++++++++++- tests/hooks/evidence-harness.test.mjs | 17 ++++++ tests/hooks/handoff-evidence.test.mjs | 45 +++++++++++++- tests/hooks/holdout-measurement.test.mjs | 16 +++++ tests/hooks/injection.test.mjs | 5 +- tests/hooks/standing-rules.test.mjs | 17 +++--- 47 files changed, 922 insertions(+), 354 deletions(-) diff --git a/evals/fixtures/mistake-transfer/scripts/audit.mjs b/evals/fixtures/mistake-transfer/scripts/audit.mjs index 7a11ad848..8d07c5abc 100644 --- a/evals/fixtures/mistake-transfer/scripts/audit.mjs +++ b/evals/fixtures/mistake-transfer/scripts/audit.mjs @@ -4,6 +4,20 @@ import { dirname } from 'node:path'; export function audit(event) { const path = process.env.TOKEN_OPTIMIZER_EVAL_AUDIT; if (!path) return; + const writerId = process.env.TOKEN_OPTIMIZER_EVAL_WRITER_ID; + const scenarioId = process.env.TOKEN_OPTIMIZER_EVAL_SCENARIO_ID; + const phase = process.env.TOKEN_OPTIMIZER_EVAL_PHASE; + if (!writerId || !scenarioId || !phase) { + throw new Error('evaluation audit requires writer, scenario, and phase provenance'); + } mkdirSync(dirname(path), { recursive: true }); - appendFileSync(path, `${JSON.stringify({ at: Date.now(), ...event })}\n`); + // Runner-owned provenance and time are authoritative. Fixture code must not + // be able to spoof another writer or move an event across the causal order. + appendFileSync(path, `${JSON.stringify({ + ...event, + writerId, + scenarioId, + phase, + at: Date.now(), + })}\n`); } diff --git a/hooks-core/adapter.mjs b/hooks-core/adapter.mjs index 9ac524c5f..55e4f644b 100644 --- a/hooks-core/adapter.mjs +++ b/hooks-core/adapter.mjs @@ -435,7 +435,7 @@ export async function run(clientName, event) { // history even though this compact index never renders a diff. const graph = load(dir); const episode = episodeMeta({ client: clientName, raw }); - const rules = standingRules(dir, graph); + const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); const index = sessionIndex(dir, graph, { episode }); if (index) parts.push(index); diff --git a/hooks-core/inject.mjs b/hooks-core/inject.mjs index c57541f2a..7efeb7923 100644 --- a/hooks-core/inject.mjs +++ b/hooks-core/inject.mjs @@ -777,16 +777,24 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph, { episode = {} } = {}) { +export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); + const relevant = new Set(relevantFindingIds); + // SessionStart has no task signal. Fail closed for situational findings until + // a client supplies ids selected from an actual prompt/tool relevance pass. + // Universal human/pinned rules are delivered separately by `standingRules`. + if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. - const findings = [...graph.nodes.values()] + const allFindings = [...graph.nodes.values()] + .filter((n) => n.kind === 'finding'); + const findings = allFindings .filter((n) => - n.kind === 'finding' - && !n.retired + !n.retired && typeof n.claim === 'string' + && typeof n.key === 'string' + && relevant.has(n.key) // These are already rendered in full by `standingRules`; listing them a // second time spends tokens without adding information. && n.pinned !== true @@ -804,7 +812,7 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { // Staleness checks can touch disk, so bound the candidates BEFORE serving // them rather than checking an arbitrarily large graph on SessionStart. const candidate = [...selected, finding]; - const preview = renderSessionIndex(findings.length, candidate); + const preview = renderSessionIndex(allFindings.length, candidate); if (estimate(preview) > budget) break; selected.push(finding); } @@ -817,11 +825,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { if (!served.length) return null; // A stale marker is longer than the fresh preview used for candidate // selection. Trim from the lowest-ranked end until the ACTUAL message fits. - while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + while (served.length && estimate(renderSessionIndex(allFindings.length, served)) > budget) { served.pop(); } if (!served.length) return null; - const text = renderSessionIndex(findings.length, served); + const text = renderSessionIndex(allFindings.length, served); const spent = estimate(text); const findingIds = served.map((finding) => finding.key); @@ -855,9 +863,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { function renderSessionIndex(total, findings) { const lines = findings.map((finding) => { - const freshness = finding.stale + const freshness = finding.stale && finding.staleEvidence ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` - : ''; + : finding.stale + ? ' [possibly stale; verify before use]' + : ''; return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; }); return `# Project wiki (${total} findings, ${findings.length} listed) @@ -893,7 +903,7 @@ ${lines.join('\n')}`; * is the one that already happened here: an always-on block that grows until it * is wallpaper, and the model stops reading the thing it always sees. */ -export function standingRules(dir, graph, { budget = standingBudget() } = {}) { +export function standingRules(dir, graph, { budget = standingBudget(), episode = {} } = {}) { const rules = [...graph.nodes.values()].filter( (n) => n.kind === 'finding' && @@ -951,6 +961,7 @@ export function standingRules(dir, graph, { budget = standingBudget() } = {}) { // measurement shows no sign of it. const truncated = noticeFor(dropped); record(dir, { + ...episode, kind: 'standing', count: lines.length, dropped, diff --git a/hooks-core/metrics.mjs b/hooks-core/metrics.mjs index e2ead5c34..7e29b2c4c 100644 --- a/hooks-core/metrics.mjs +++ b/hooks-core/metrics.mjs @@ -759,8 +759,12 @@ export function report(dir) { // file 95 historical command injections into the file-read balance -- exactly // the dilution the split exists to prevent. const isCommand = (e) => e.surface === 'command' || e.trigger === 'command'; - const commandInjections = allInjections.filter(isCommand); - const injections = allInjections.filter((e) => !isCommand(e)); + // Session-start delivery has no file anchor or downstream read join either. + // Report its cost independently so it cannot dilute the file-touch estimate. + const isSessionStart = (e) => e.surface === 'session-start'; + const sessionStartInjections = allInjections.filter(isSessionStart); + const commandInjections = allInjections.filter((e) => !isSessionStart(e) && isCommand(e)); + const injections = allInjections.filter((e) => !isSessionStart(e) && !isCommand(e)); const treated = injections.filter((e) => !e.holdout); const withheld = injections.filter((e) => e.holdout); @@ -838,6 +842,10 @@ export function report(dir) { return { injections: treated.length, + sessionStartInjections: sessionStartInjections.length, + sessionStartInjectedTokens: sessionStartInjections.reduce( + (sum, event) => sum + (event.deliveredTokens ?? event.tokens ?? 0), 0 + ), commandInjections: commandInjections.length, commandHoldouts: commandInjections.filter((e) => e.holdout).length, holdouts: withheld.length, @@ -1078,7 +1086,9 @@ function matchesFilters(event, filters) { if (!filters[field]) continue; let actual = event[field]; if (event.kind === 'handoff-run') { - actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + actual = field === 'taskId' + ? event.scenarioId + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); } @@ -1096,6 +1106,10 @@ function handoffArmMetrics(runs) { ); return { runs: runs.length, + delivery: proportionInterval( + runs.filter((run) => run.delivery?.delivered === true).length, + runs.length + ), correctness: proportionInterval(count('correct'), runs.length), firstPass: proportionInterval(count('firstPass'), runs.length), mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), @@ -1144,6 +1158,10 @@ const handoffKey = (run) => [ run.scenarioId || 'unknown', ].join('|'); +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; +const handoffMinimumPairs = () => + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + function handoffCohorts(runs) { const grouped = new Map(); for (const run of runs) { @@ -1151,7 +1169,7 @@ function handoffCohorts(runs) { if (!grouped.has(key)) grouped.set(key, []); grouped.get(key).push(run); } - const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + const minimumPairs = handoffMinimumPairs(); return [...grouped.entries()].map(([key, rows]) => { const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), @@ -1167,14 +1185,18 @@ function handoffCohorts(runs) { : null; const emptyRecurrence = arms.empty.mistakeExecuted.rate; const naturalRecurrence = arms.natural.mistakeExecuted.rate; - const relativeRecurrenceReduction = emptyRecurrence + const relativeRecurrenceReduction = emptyRecurrence !== null + && naturalRecurrence !== null + && emptyRecurrence !== 0 ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence : null; const controlsSafe = ['irrelevant', 'stale'].every((arm) => - arms[arm].correctness.rate === null - || arms.empty.correctness.rate === null - || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + arms[arm].correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 ); + const irrelevantSuppressed = arms.irrelevant.delivery.rate !== null + && arms.irrelevant.delivery.rate === 0; const gates = { minimumPairs: naturalVsEmpty.pairs >= minimumPairs, capture: captureRate !== null && captureRate >= 0.8, @@ -1182,9 +1204,11 @@ function handoffCohorts(runs) { relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, correctness: - (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + arms.natural.correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms.natural.correctness.rate >= arms.empty.correctness.rate - 0.1, preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, - negativeControls: controlsSafe, + negativeControls: controlsSafe && irrelevantSuppressed, }; const claimReady = Object.values(gates).every(Boolean); return { @@ -1207,8 +1231,6 @@ function handoffCohorts(runs) { }); } -const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; - function concurrencySummary(runs) { const natural = runs.filter((run) => run.arm === 'natural'); const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); @@ -1343,8 +1365,7 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, - handoffMinimumPairs: - Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), + handoffMinimumPairs: handoffMinimumPairs(), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/cline/hooks/token-optimizer/lib/adapter.mjs b/integrations/cline/hooks/token-optimizer/lib/adapter.mjs index c99d512cc..0e079eca7 100644 --- a/integrations/cline/hooks/token-optimizer/lib/adapter.mjs +++ b/integrations/cline/hooks/token-optimizer/lib/adapter.mjs @@ -437,7 +437,7 @@ export async function run(clientName, event) { // history even though this compact index never renders a diff. const graph = load(dir); const episode = episodeMeta({ client: clientName, raw }); - const rules = standingRules(dir, graph); + const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); const index = sessionIndex(dir, graph, { episode }); if (index) parts.push(index); diff --git a/integrations/cline/hooks/token-optimizer/lib/inject.mjs b/integrations/cline/hooks/token-optimizer/lib/inject.mjs index db1ecec4a..1ec5b4ce4 100644 --- a/integrations/cline/hooks/token-optimizer/lib/inject.mjs +++ b/integrations/cline/hooks/token-optimizer/lib/inject.mjs @@ -779,16 +779,24 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph, { episode = {} } = {}) { +export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); + const relevant = new Set(relevantFindingIds); + // SessionStart has no task signal. Fail closed for situational findings until + // a client supplies ids selected from an actual prompt/tool relevance pass. + // Universal human/pinned rules are delivered separately by `standingRules`. + if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. - const findings = [...graph.nodes.values()] + const allFindings = [...graph.nodes.values()] + .filter((n) => n.kind === 'finding'); + const findings = allFindings .filter((n) => - n.kind === 'finding' - && !n.retired + !n.retired && typeof n.claim === 'string' + && typeof n.key === 'string' + && relevant.has(n.key) // These are already rendered in full by `standingRules`; listing them a // second time spends tokens without adding information. && n.pinned !== true @@ -806,7 +814,7 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { // Staleness checks can touch disk, so bound the candidates BEFORE serving // them rather than checking an arbitrarily large graph on SessionStart. const candidate = [...selected, finding]; - const preview = renderSessionIndex(findings.length, candidate); + const preview = renderSessionIndex(allFindings.length, candidate); if (estimate(preview) > budget) break; selected.push(finding); } @@ -819,11 +827,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { if (!served.length) return null; // A stale marker is longer than the fresh preview used for candidate // selection. Trim from the lowest-ranked end until the ACTUAL message fits. - while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + while (served.length && estimate(renderSessionIndex(allFindings.length, served)) > budget) { served.pop(); } if (!served.length) return null; - const text = renderSessionIndex(findings.length, served); + const text = renderSessionIndex(allFindings.length, served); const spent = estimate(text); const findingIds = served.map((finding) => finding.key); @@ -857,9 +865,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { function renderSessionIndex(total, findings) { const lines = findings.map((finding) => { - const freshness = finding.stale + const freshness = finding.stale && finding.staleEvidence ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` - : ''; + : finding.stale + ? ' [possibly stale; verify before use]' + : ''; return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; }); return `# Project wiki (${total} findings, ${findings.length} listed) @@ -895,7 +905,7 @@ ${lines.join('\n')}`; * is the one that already happened here: an always-on block that grows until it * is wallpaper, and the model stops reading the thing it always sees. */ -export function standingRules(dir, graph, { budget = standingBudget() } = {}) { +export function standingRules(dir, graph, { budget = standingBudget(), episode = {} } = {}) { const rules = [...graph.nodes.values()].filter( (n) => n.kind === 'finding' && @@ -953,6 +963,7 @@ export function standingRules(dir, graph, { budget = standingBudget() } = {}) { // measurement shows no sign of it. const truncated = noticeFor(dropped); record(dir, { + ...episode, kind: 'standing', count: lines.length, dropped, diff --git a/integrations/cline/hooks/token-optimizer/lib/metrics.mjs b/integrations/cline/hooks/token-optimizer/lib/metrics.mjs index 3e4c4bb59..f6ab65780 100644 --- a/integrations/cline/hooks/token-optimizer/lib/metrics.mjs +++ b/integrations/cline/hooks/token-optimizer/lib/metrics.mjs @@ -761,8 +761,12 @@ export function report(dir) { // file 95 historical command injections into the file-read balance -- exactly // the dilution the split exists to prevent. const isCommand = (e) => e.surface === 'command' || e.trigger === 'command'; - const commandInjections = allInjections.filter(isCommand); - const injections = allInjections.filter((e) => !isCommand(e)); + // Session-start delivery has no file anchor or downstream read join either. + // Report its cost independently so it cannot dilute the file-touch estimate. + const isSessionStart = (e) => e.surface === 'session-start'; + const sessionStartInjections = allInjections.filter(isSessionStart); + const commandInjections = allInjections.filter((e) => !isSessionStart(e) && isCommand(e)); + const injections = allInjections.filter((e) => !isSessionStart(e) && !isCommand(e)); const treated = injections.filter((e) => !e.holdout); const withheld = injections.filter((e) => e.holdout); @@ -840,6 +844,10 @@ export function report(dir) { return { injections: treated.length, + sessionStartInjections: sessionStartInjections.length, + sessionStartInjectedTokens: sessionStartInjections.reduce( + (sum, event) => sum + (event.deliveredTokens ?? event.tokens ?? 0), 0 + ), commandInjections: commandInjections.length, commandHoldouts: commandInjections.filter((e) => e.holdout).length, holdouts: withheld.length, @@ -1080,7 +1088,9 @@ function matchesFilters(event, filters) { if (!filters[field]) continue; let actual = event[field]; if (event.kind === 'handoff-run') { - actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + actual = field === 'taskId' + ? event.scenarioId + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); } @@ -1098,6 +1108,10 @@ function handoffArmMetrics(runs) { ); return { runs: runs.length, + delivery: proportionInterval( + runs.filter((run) => run.delivery?.delivered === true).length, + runs.length + ), correctness: proportionInterval(count('correct'), runs.length), firstPass: proportionInterval(count('firstPass'), runs.length), mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), @@ -1146,6 +1160,10 @@ const handoffKey = (run) => [ run.scenarioId || 'unknown', ].join('|'); +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; +const handoffMinimumPairs = () => + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + function handoffCohorts(runs) { const grouped = new Map(); for (const run of runs) { @@ -1153,7 +1171,7 @@ function handoffCohorts(runs) { if (!grouped.has(key)) grouped.set(key, []); grouped.get(key).push(run); } - const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + const minimumPairs = handoffMinimumPairs(); return [...grouped.entries()].map(([key, rows]) => { const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), @@ -1169,14 +1187,18 @@ function handoffCohorts(runs) { : null; const emptyRecurrence = arms.empty.mistakeExecuted.rate; const naturalRecurrence = arms.natural.mistakeExecuted.rate; - const relativeRecurrenceReduction = emptyRecurrence + const relativeRecurrenceReduction = emptyRecurrence !== null + && naturalRecurrence !== null + && emptyRecurrence !== 0 ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence : null; const controlsSafe = ['irrelevant', 'stale'].every((arm) => - arms[arm].correctness.rate === null - || arms.empty.correctness.rate === null - || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + arms[arm].correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 ); + const irrelevantSuppressed = arms.irrelevant.delivery.rate !== null + && arms.irrelevant.delivery.rate === 0; const gates = { minimumPairs: naturalVsEmpty.pairs >= minimumPairs, capture: captureRate !== null && captureRate >= 0.8, @@ -1184,9 +1206,11 @@ function handoffCohorts(runs) { relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, correctness: - (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + arms.natural.correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms.natural.correctness.rate >= arms.empty.correctness.rate - 0.1, preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, - negativeControls: controlsSafe, + negativeControls: controlsSafe && irrelevantSuppressed, }; const claimReady = Object.values(gates).every(Boolean); return { @@ -1209,8 +1233,6 @@ function handoffCohorts(runs) { }); } -const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; - function concurrencySummary(runs) { const natural = runs.filter((run) => run.arm === 'natural'); const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); @@ -1345,8 +1367,7 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, - handoffMinimumPairs: - Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), + handoffMinimumPairs: handoffMinimumPairs(), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/codex/hooks/lib/adapter.mjs b/integrations/codex/hooks/lib/adapter.mjs index c99d512cc..0e079eca7 100644 --- a/integrations/codex/hooks/lib/adapter.mjs +++ b/integrations/codex/hooks/lib/adapter.mjs @@ -437,7 +437,7 @@ export async function run(clientName, event) { // history even though this compact index never renders a diff. const graph = load(dir); const episode = episodeMeta({ client: clientName, raw }); - const rules = standingRules(dir, graph); + const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); const index = sessionIndex(dir, graph, { episode }); if (index) parts.push(index); diff --git a/integrations/codex/hooks/lib/inject.mjs b/integrations/codex/hooks/lib/inject.mjs index db1ecec4a..1ec5b4ce4 100644 --- a/integrations/codex/hooks/lib/inject.mjs +++ b/integrations/codex/hooks/lib/inject.mjs @@ -779,16 +779,24 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph, { episode = {} } = {}) { +export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); + const relevant = new Set(relevantFindingIds); + // SessionStart has no task signal. Fail closed for situational findings until + // a client supplies ids selected from an actual prompt/tool relevance pass. + // Universal human/pinned rules are delivered separately by `standingRules`. + if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. - const findings = [...graph.nodes.values()] + const allFindings = [...graph.nodes.values()] + .filter((n) => n.kind === 'finding'); + const findings = allFindings .filter((n) => - n.kind === 'finding' - && !n.retired + !n.retired && typeof n.claim === 'string' + && typeof n.key === 'string' + && relevant.has(n.key) // These are already rendered in full by `standingRules`; listing them a // second time spends tokens without adding information. && n.pinned !== true @@ -806,7 +814,7 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { // Staleness checks can touch disk, so bound the candidates BEFORE serving // them rather than checking an arbitrarily large graph on SessionStart. const candidate = [...selected, finding]; - const preview = renderSessionIndex(findings.length, candidate); + const preview = renderSessionIndex(allFindings.length, candidate); if (estimate(preview) > budget) break; selected.push(finding); } @@ -819,11 +827,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { if (!served.length) return null; // A stale marker is longer than the fresh preview used for candidate // selection. Trim from the lowest-ranked end until the ACTUAL message fits. - while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + while (served.length && estimate(renderSessionIndex(allFindings.length, served)) > budget) { served.pop(); } if (!served.length) return null; - const text = renderSessionIndex(findings.length, served); + const text = renderSessionIndex(allFindings.length, served); const spent = estimate(text); const findingIds = served.map((finding) => finding.key); @@ -857,9 +865,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { function renderSessionIndex(total, findings) { const lines = findings.map((finding) => { - const freshness = finding.stale + const freshness = finding.stale && finding.staleEvidence ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` - : ''; + : finding.stale + ? ' [possibly stale; verify before use]' + : ''; return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; }); return `# Project wiki (${total} findings, ${findings.length} listed) @@ -895,7 +905,7 @@ ${lines.join('\n')}`; * is the one that already happened here: an always-on block that grows until it * is wallpaper, and the model stops reading the thing it always sees. */ -export function standingRules(dir, graph, { budget = standingBudget() } = {}) { +export function standingRules(dir, graph, { budget = standingBudget(), episode = {} } = {}) { const rules = [...graph.nodes.values()].filter( (n) => n.kind === 'finding' && @@ -953,6 +963,7 @@ export function standingRules(dir, graph, { budget = standingBudget() } = {}) { // measurement shows no sign of it. const truncated = noticeFor(dropped); record(dir, { + ...episode, kind: 'standing', count: lines.length, dropped, diff --git a/integrations/codex/hooks/lib/metrics.mjs b/integrations/codex/hooks/lib/metrics.mjs index 3e4c4bb59..f6ab65780 100644 --- a/integrations/codex/hooks/lib/metrics.mjs +++ b/integrations/codex/hooks/lib/metrics.mjs @@ -761,8 +761,12 @@ export function report(dir) { // file 95 historical command injections into the file-read balance -- exactly // the dilution the split exists to prevent. const isCommand = (e) => e.surface === 'command' || e.trigger === 'command'; - const commandInjections = allInjections.filter(isCommand); - const injections = allInjections.filter((e) => !isCommand(e)); + // Session-start delivery has no file anchor or downstream read join either. + // Report its cost independently so it cannot dilute the file-touch estimate. + const isSessionStart = (e) => e.surface === 'session-start'; + const sessionStartInjections = allInjections.filter(isSessionStart); + const commandInjections = allInjections.filter((e) => !isSessionStart(e) && isCommand(e)); + const injections = allInjections.filter((e) => !isSessionStart(e) && !isCommand(e)); const treated = injections.filter((e) => !e.holdout); const withheld = injections.filter((e) => e.holdout); @@ -840,6 +844,10 @@ export function report(dir) { return { injections: treated.length, + sessionStartInjections: sessionStartInjections.length, + sessionStartInjectedTokens: sessionStartInjections.reduce( + (sum, event) => sum + (event.deliveredTokens ?? event.tokens ?? 0), 0 + ), commandInjections: commandInjections.length, commandHoldouts: commandInjections.filter((e) => e.holdout).length, holdouts: withheld.length, @@ -1080,7 +1088,9 @@ function matchesFilters(event, filters) { if (!filters[field]) continue; let actual = event[field]; if (event.kind === 'handoff-run') { - actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + actual = field === 'taskId' + ? event.scenarioId + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); } @@ -1098,6 +1108,10 @@ function handoffArmMetrics(runs) { ); return { runs: runs.length, + delivery: proportionInterval( + runs.filter((run) => run.delivery?.delivered === true).length, + runs.length + ), correctness: proportionInterval(count('correct'), runs.length), firstPass: proportionInterval(count('firstPass'), runs.length), mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), @@ -1146,6 +1160,10 @@ const handoffKey = (run) => [ run.scenarioId || 'unknown', ].join('|'); +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; +const handoffMinimumPairs = () => + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + function handoffCohorts(runs) { const grouped = new Map(); for (const run of runs) { @@ -1153,7 +1171,7 @@ function handoffCohorts(runs) { if (!grouped.has(key)) grouped.set(key, []); grouped.get(key).push(run); } - const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + const minimumPairs = handoffMinimumPairs(); return [...grouped.entries()].map(([key, rows]) => { const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), @@ -1169,14 +1187,18 @@ function handoffCohorts(runs) { : null; const emptyRecurrence = arms.empty.mistakeExecuted.rate; const naturalRecurrence = arms.natural.mistakeExecuted.rate; - const relativeRecurrenceReduction = emptyRecurrence + const relativeRecurrenceReduction = emptyRecurrence !== null + && naturalRecurrence !== null + && emptyRecurrence !== 0 ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence : null; const controlsSafe = ['irrelevant', 'stale'].every((arm) => - arms[arm].correctness.rate === null - || arms.empty.correctness.rate === null - || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + arms[arm].correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 ); + const irrelevantSuppressed = arms.irrelevant.delivery.rate !== null + && arms.irrelevant.delivery.rate === 0; const gates = { minimumPairs: naturalVsEmpty.pairs >= minimumPairs, capture: captureRate !== null && captureRate >= 0.8, @@ -1184,9 +1206,11 @@ function handoffCohorts(runs) { relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, correctness: - (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + arms.natural.correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms.natural.correctness.rate >= arms.empty.correctness.rate - 0.1, preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, - negativeControls: controlsSafe, + negativeControls: controlsSafe && irrelevantSuppressed, }; const claimReady = Object.values(gates).every(Boolean); return { @@ -1209,8 +1233,6 @@ function handoffCohorts(runs) { }); } -const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; - function concurrencySummary(runs) { const natural = runs.filter((run) => run.arm === 'natural'); const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); @@ -1345,8 +1367,7 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, - handoffMinimumPairs: - Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), + handoffMinimumPairs: handoffMinimumPairs(), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/codex/plugin/hooks/lib/adapter.mjs b/integrations/codex/plugin/hooks/lib/adapter.mjs index c99d512cc..0e079eca7 100644 --- a/integrations/codex/plugin/hooks/lib/adapter.mjs +++ b/integrations/codex/plugin/hooks/lib/adapter.mjs @@ -437,7 +437,7 @@ export async function run(clientName, event) { // history even though this compact index never renders a diff. const graph = load(dir); const episode = episodeMeta({ client: clientName, raw }); - const rules = standingRules(dir, graph); + const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); const index = sessionIndex(dir, graph, { episode }); if (index) parts.push(index); diff --git a/integrations/codex/plugin/hooks/lib/inject.mjs b/integrations/codex/plugin/hooks/lib/inject.mjs index db1ecec4a..1ec5b4ce4 100644 --- a/integrations/codex/plugin/hooks/lib/inject.mjs +++ b/integrations/codex/plugin/hooks/lib/inject.mjs @@ -779,16 +779,24 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph, { episode = {} } = {}) { +export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); + const relevant = new Set(relevantFindingIds); + // SessionStart has no task signal. Fail closed for situational findings until + // a client supplies ids selected from an actual prompt/tool relevance pass. + // Universal human/pinned rules are delivered separately by `standingRules`. + if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. - const findings = [...graph.nodes.values()] + const allFindings = [...graph.nodes.values()] + .filter((n) => n.kind === 'finding'); + const findings = allFindings .filter((n) => - n.kind === 'finding' - && !n.retired + !n.retired && typeof n.claim === 'string' + && typeof n.key === 'string' + && relevant.has(n.key) // These are already rendered in full by `standingRules`; listing them a // second time spends tokens without adding information. && n.pinned !== true @@ -806,7 +814,7 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { // Staleness checks can touch disk, so bound the candidates BEFORE serving // them rather than checking an arbitrarily large graph on SessionStart. const candidate = [...selected, finding]; - const preview = renderSessionIndex(findings.length, candidate); + const preview = renderSessionIndex(allFindings.length, candidate); if (estimate(preview) > budget) break; selected.push(finding); } @@ -819,11 +827,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { if (!served.length) return null; // A stale marker is longer than the fresh preview used for candidate // selection. Trim from the lowest-ranked end until the ACTUAL message fits. - while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + while (served.length && estimate(renderSessionIndex(allFindings.length, served)) > budget) { served.pop(); } if (!served.length) return null; - const text = renderSessionIndex(findings.length, served); + const text = renderSessionIndex(allFindings.length, served); const spent = estimate(text); const findingIds = served.map((finding) => finding.key); @@ -857,9 +865,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { function renderSessionIndex(total, findings) { const lines = findings.map((finding) => { - const freshness = finding.stale + const freshness = finding.stale && finding.staleEvidence ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` - : ''; + : finding.stale + ? ' [possibly stale; verify before use]' + : ''; return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; }); return `# Project wiki (${total} findings, ${findings.length} listed) @@ -895,7 +905,7 @@ ${lines.join('\n')}`; * is the one that already happened here: an always-on block that grows until it * is wallpaper, and the model stops reading the thing it always sees. */ -export function standingRules(dir, graph, { budget = standingBudget() } = {}) { +export function standingRules(dir, graph, { budget = standingBudget(), episode = {} } = {}) { const rules = [...graph.nodes.values()].filter( (n) => n.kind === 'finding' && @@ -953,6 +963,7 @@ export function standingRules(dir, graph, { budget = standingBudget() } = {}) { // measurement shows no sign of it. const truncated = noticeFor(dropped); record(dir, { + ...episode, kind: 'standing', count: lines.length, dropped, diff --git a/integrations/codex/plugin/hooks/lib/metrics.mjs b/integrations/codex/plugin/hooks/lib/metrics.mjs index 3e4c4bb59..f6ab65780 100644 --- a/integrations/codex/plugin/hooks/lib/metrics.mjs +++ b/integrations/codex/plugin/hooks/lib/metrics.mjs @@ -761,8 +761,12 @@ export function report(dir) { // file 95 historical command injections into the file-read balance -- exactly // the dilution the split exists to prevent. const isCommand = (e) => e.surface === 'command' || e.trigger === 'command'; - const commandInjections = allInjections.filter(isCommand); - const injections = allInjections.filter((e) => !isCommand(e)); + // Session-start delivery has no file anchor or downstream read join either. + // Report its cost independently so it cannot dilute the file-touch estimate. + const isSessionStart = (e) => e.surface === 'session-start'; + const sessionStartInjections = allInjections.filter(isSessionStart); + const commandInjections = allInjections.filter((e) => !isSessionStart(e) && isCommand(e)); + const injections = allInjections.filter((e) => !isSessionStart(e) && !isCommand(e)); const treated = injections.filter((e) => !e.holdout); const withheld = injections.filter((e) => e.holdout); @@ -840,6 +844,10 @@ export function report(dir) { return { injections: treated.length, + sessionStartInjections: sessionStartInjections.length, + sessionStartInjectedTokens: sessionStartInjections.reduce( + (sum, event) => sum + (event.deliveredTokens ?? event.tokens ?? 0), 0 + ), commandInjections: commandInjections.length, commandHoldouts: commandInjections.filter((e) => e.holdout).length, holdouts: withheld.length, @@ -1080,7 +1088,9 @@ function matchesFilters(event, filters) { if (!filters[field]) continue; let actual = event[field]; if (event.kind === 'handoff-run') { - actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + actual = field === 'taskId' + ? event.scenarioId + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); } @@ -1098,6 +1108,10 @@ function handoffArmMetrics(runs) { ); return { runs: runs.length, + delivery: proportionInterval( + runs.filter((run) => run.delivery?.delivered === true).length, + runs.length + ), correctness: proportionInterval(count('correct'), runs.length), firstPass: proportionInterval(count('firstPass'), runs.length), mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), @@ -1146,6 +1160,10 @@ const handoffKey = (run) => [ run.scenarioId || 'unknown', ].join('|'); +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; +const handoffMinimumPairs = () => + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + function handoffCohorts(runs) { const grouped = new Map(); for (const run of runs) { @@ -1153,7 +1171,7 @@ function handoffCohorts(runs) { if (!grouped.has(key)) grouped.set(key, []); grouped.get(key).push(run); } - const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + const minimumPairs = handoffMinimumPairs(); return [...grouped.entries()].map(([key, rows]) => { const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), @@ -1169,14 +1187,18 @@ function handoffCohorts(runs) { : null; const emptyRecurrence = arms.empty.mistakeExecuted.rate; const naturalRecurrence = arms.natural.mistakeExecuted.rate; - const relativeRecurrenceReduction = emptyRecurrence + const relativeRecurrenceReduction = emptyRecurrence !== null + && naturalRecurrence !== null + && emptyRecurrence !== 0 ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence : null; const controlsSafe = ['irrelevant', 'stale'].every((arm) => - arms[arm].correctness.rate === null - || arms.empty.correctness.rate === null - || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + arms[arm].correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 ); + const irrelevantSuppressed = arms.irrelevant.delivery.rate !== null + && arms.irrelevant.delivery.rate === 0; const gates = { minimumPairs: naturalVsEmpty.pairs >= minimumPairs, capture: captureRate !== null && captureRate >= 0.8, @@ -1184,9 +1206,11 @@ function handoffCohorts(runs) { relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, correctness: - (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + arms.natural.correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms.natural.correctness.rate >= arms.empty.correctness.rate - 0.1, preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, - negativeControls: controlsSafe, + negativeControls: controlsSafe && irrelevantSuppressed, }; const claimReady = Object.values(gates).every(Boolean); return { @@ -1209,8 +1233,6 @@ function handoffCohorts(runs) { }); } -const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; - function concurrencySummary(runs) { const natural = runs.filter((run) => run.arm === 'natural'); const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); @@ -1345,8 +1367,7 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, - handoffMinimumPairs: - Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), + handoffMinimumPairs: handoffMinimumPairs(), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/copilot/.github/hooks/lib/adapter.mjs b/integrations/copilot/.github/hooks/lib/adapter.mjs index c99d512cc..0e079eca7 100644 --- a/integrations/copilot/.github/hooks/lib/adapter.mjs +++ b/integrations/copilot/.github/hooks/lib/adapter.mjs @@ -437,7 +437,7 @@ export async function run(clientName, event) { // history even though this compact index never renders a diff. const graph = load(dir); const episode = episodeMeta({ client: clientName, raw }); - const rules = standingRules(dir, graph); + const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); const index = sessionIndex(dir, graph, { episode }); if (index) parts.push(index); diff --git a/integrations/copilot/.github/hooks/lib/inject.mjs b/integrations/copilot/.github/hooks/lib/inject.mjs index db1ecec4a..1ec5b4ce4 100644 --- a/integrations/copilot/.github/hooks/lib/inject.mjs +++ b/integrations/copilot/.github/hooks/lib/inject.mjs @@ -779,16 +779,24 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph, { episode = {} } = {}) { +export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); + const relevant = new Set(relevantFindingIds); + // SessionStart has no task signal. Fail closed for situational findings until + // a client supplies ids selected from an actual prompt/tool relevance pass. + // Universal human/pinned rules are delivered separately by `standingRules`. + if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. - const findings = [...graph.nodes.values()] + const allFindings = [...graph.nodes.values()] + .filter((n) => n.kind === 'finding'); + const findings = allFindings .filter((n) => - n.kind === 'finding' - && !n.retired + !n.retired && typeof n.claim === 'string' + && typeof n.key === 'string' + && relevant.has(n.key) // These are already rendered in full by `standingRules`; listing them a // second time spends tokens without adding information. && n.pinned !== true @@ -806,7 +814,7 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { // Staleness checks can touch disk, so bound the candidates BEFORE serving // them rather than checking an arbitrarily large graph on SessionStart. const candidate = [...selected, finding]; - const preview = renderSessionIndex(findings.length, candidate); + const preview = renderSessionIndex(allFindings.length, candidate); if (estimate(preview) > budget) break; selected.push(finding); } @@ -819,11 +827,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { if (!served.length) return null; // A stale marker is longer than the fresh preview used for candidate // selection. Trim from the lowest-ranked end until the ACTUAL message fits. - while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + while (served.length && estimate(renderSessionIndex(allFindings.length, served)) > budget) { served.pop(); } if (!served.length) return null; - const text = renderSessionIndex(findings.length, served); + const text = renderSessionIndex(allFindings.length, served); const spent = estimate(text); const findingIds = served.map((finding) => finding.key); @@ -857,9 +865,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { function renderSessionIndex(total, findings) { const lines = findings.map((finding) => { - const freshness = finding.stale + const freshness = finding.stale && finding.staleEvidence ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` - : ''; + : finding.stale + ? ' [possibly stale; verify before use]' + : ''; return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; }); return `# Project wiki (${total} findings, ${findings.length} listed) @@ -895,7 +905,7 @@ ${lines.join('\n')}`; * is the one that already happened here: an always-on block that grows until it * is wallpaper, and the model stops reading the thing it always sees. */ -export function standingRules(dir, graph, { budget = standingBudget() } = {}) { +export function standingRules(dir, graph, { budget = standingBudget(), episode = {} } = {}) { const rules = [...graph.nodes.values()].filter( (n) => n.kind === 'finding' && @@ -953,6 +963,7 @@ export function standingRules(dir, graph, { budget = standingBudget() } = {}) { // measurement shows no sign of it. const truncated = noticeFor(dropped); record(dir, { + ...episode, kind: 'standing', count: lines.length, dropped, diff --git a/integrations/copilot/.github/hooks/lib/metrics.mjs b/integrations/copilot/.github/hooks/lib/metrics.mjs index 3e4c4bb59..f6ab65780 100644 --- a/integrations/copilot/.github/hooks/lib/metrics.mjs +++ b/integrations/copilot/.github/hooks/lib/metrics.mjs @@ -761,8 +761,12 @@ export function report(dir) { // file 95 historical command injections into the file-read balance -- exactly // the dilution the split exists to prevent. const isCommand = (e) => e.surface === 'command' || e.trigger === 'command'; - const commandInjections = allInjections.filter(isCommand); - const injections = allInjections.filter((e) => !isCommand(e)); + // Session-start delivery has no file anchor or downstream read join either. + // Report its cost independently so it cannot dilute the file-touch estimate. + const isSessionStart = (e) => e.surface === 'session-start'; + const sessionStartInjections = allInjections.filter(isSessionStart); + const commandInjections = allInjections.filter((e) => !isSessionStart(e) && isCommand(e)); + const injections = allInjections.filter((e) => !isSessionStart(e) && !isCommand(e)); const treated = injections.filter((e) => !e.holdout); const withheld = injections.filter((e) => e.holdout); @@ -840,6 +844,10 @@ export function report(dir) { return { injections: treated.length, + sessionStartInjections: sessionStartInjections.length, + sessionStartInjectedTokens: sessionStartInjections.reduce( + (sum, event) => sum + (event.deliveredTokens ?? event.tokens ?? 0), 0 + ), commandInjections: commandInjections.length, commandHoldouts: commandInjections.filter((e) => e.holdout).length, holdouts: withheld.length, @@ -1080,7 +1088,9 @@ function matchesFilters(event, filters) { if (!filters[field]) continue; let actual = event[field]; if (event.kind === 'handoff-run') { - actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + actual = field === 'taskId' + ? event.scenarioId + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); } @@ -1098,6 +1108,10 @@ function handoffArmMetrics(runs) { ); return { runs: runs.length, + delivery: proportionInterval( + runs.filter((run) => run.delivery?.delivered === true).length, + runs.length + ), correctness: proportionInterval(count('correct'), runs.length), firstPass: proportionInterval(count('firstPass'), runs.length), mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), @@ -1146,6 +1160,10 @@ const handoffKey = (run) => [ run.scenarioId || 'unknown', ].join('|'); +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; +const handoffMinimumPairs = () => + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + function handoffCohorts(runs) { const grouped = new Map(); for (const run of runs) { @@ -1153,7 +1171,7 @@ function handoffCohorts(runs) { if (!grouped.has(key)) grouped.set(key, []); grouped.get(key).push(run); } - const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + const minimumPairs = handoffMinimumPairs(); return [...grouped.entries()].map(([key, rows]) => { const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), @@ -1169,14 +1187,18 @@ function handoffCohorts(runs) { : null; const emptyRecurrence = arms.empty.mistakeExecuted.rate; const naturalRecurrence = arms.natural.mistakeExecuted.rate; - const relativeRecurrenceReduction = emptyRecurrence + const relativeRecurrenceReduction = emptyRecurrence !== null + && naturalRecurrence !== null + && emptyRecurrence !== 0 ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence : null; const controlsSafe = ['irrelevant', 'stale'].every((arm) => - arms[arm].correctness.rate === null - || arms.empty.correctness.rate === null - || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + arms[arm].correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 ); + const irrelevantSuppressed = arms.irrelevant.delivery.rate !== null + && arms.irrelevant.delivery.rate === 0; const gates = { minimumPairs: naturalVsEmpty.pairs >= minimumPairs, capture: captureRate !== null && captureRate >= 0.8, @@ -1184,9 +1206,11 @@ function handoffCohorts(runs) { relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, correctness: - (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + arms.natural.correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms.natural.correctness.rate >= arms.empty.correctness.rate - 0.1, preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, - negativeControls: controlsSafe, + negativeControls: controlsSafe && irrelevantSuppressed, }; const claimReady = Object.values(gates).every(Boolean); return { @@ -1209,8 +1233,6 @@ function handoffCohorts(runs) { }); } -const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; - function concurrencySummary(runs) { const natural = runs.filter((run) => run.arm === 'natural'); const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); @@ -1345,8 +1367,7 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, - handoffMinimumPairs: - Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), + handoffMinimumPairs: handoffMinimumPairs(), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/cursor/hooks/lib/adapter.mjs b/integrations/cursor/hooks/lib/adapter.mjs index c99d512cc..0e079eca7 100644 --- a/integrations/cursor/hooks/lib/adapter.mjs +++ b/integrations/cursor/hooks/lib/adapter.mjs @@ -437,7 +437,7 @@ export async function run(clientName, event) { // history even though this compact index never renders a diff. const graph = load(dir); const episode = episodeMeta({ client: clientName, raw }); - const rules = standingRules(dir, graph); + const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); const index = sessionIndex(dir, graph, { episode }); if (index) parts.push(index); diff --git a/integrations/cursor/hooks/lib/inject.mjs b/integrations/cursor/hooks/lib/inject.mjs index db1ecec4a..1ec5b4ce4 100644 --- a/integrations/cursor/hooks/lib/inject.mjs +++ b/integrations/cursor/hooks/lib/inject.mjs @@ -779,16 +779,24 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph, { episode = {} } = {}) { +export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); + const relevant = new Set(relevantFindingIds); + // SessionStart has no task signal. Fail closed for situational findings until + // a client supplies ids selected from an actual prompt/tool relevance pass. + // Universal human/pinned rules are delivered separately by `standingRules`. + if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. - const findings = [...graph.nodes.values()] + const allFindings = [...graph.nodes.values()] + .filter((n) => n.kind === 'finding'); + const findings = allFindings .filter((n) => - n.kind === 'finding' - && !n.retired + !n.retired && typeof n.claim === 'string' + && typeof n.key === 'string' + && relevant.has(n.key) // These are already rendered in full by `standingRules`; listing them a // second time spends tokens without adding information. && n.pinned !== true @@ -806,7 +814,7 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { // Staleness checks can touch disk, so bound the candidates BEFORE serving // them rather than checking an arbitrarily large graph on SessionStart. const candidate = [...selected, finding]; - const preview = renderSessionIndex(findings.length, candidate); + const preview = renderSessionIndex(allFindings.length, candidate); if (estimate(preview) > budget) break; selected.push(finding); } @@ -819,11 +827,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { if (!served.length) return null; // A stale marker is longer than the fresh preview used for candidate // selection. Trim from the lowest-ranked end until the ACTUAL message fits. - while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + while (served.length && estimate(renderSessionIndex(allFindings.length, served)) > budget) { served.pop(); } if (!served.length) return null; - const text = renderSessionIndex(findings.length, served); + const text = renderSessionIndex(allFindings.length, served); const spent = estimate(text); const findingIds = served.map((finding) => finding.key); @@ -857,9 +865,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { function renderSessionIndex(total, findings) { const lines = findings.map((finding) => { - const freshness = finding.stale + const freshness = finding.stale && finding.staleEvidence ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` - : ''; + : finding.stale + ? ' [possibly stale; verify before use]' + : ''; return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; }); return `# Project wiki (${total} findings, ${findings.length} listed) @@ -895,7 +905,7 @@ ${lines.join('\n')}`; * is the one that already happened here: an always-on block that grows until it * is wallpaper, and the model stops reading the thing it always sees. */ -export function standingRules(dir, graph, { budget = standingBudget() } = {}) { +export function standingRules(dir, graph, { budget = standingBudget(), episode = {} } = {}) { const rules = [...graph.nodes.values()].filter( (n) => n.kind === 'finding' && @@ -953,6 +963,7 @@ export function standingRules(dir, graph, { budget = standingBudget() } = {}) { // measurement shows no sign of it. const truncated = noticeFor(dropped); record(dir, { + ...episode, kind: 'standing', count: lines.length, dropped, diff --git a/integrations/cursor/hooks/lib/metrics.mjs b/integrations/cursor/hooks/lib/metrics.mjs index 3e4c4bb59..f6ab65780 100644 --- a/integrations/cursor/hooks/lib/metrics.mjs +++ b/integrations/cursor/hooks/lib/metrics.mjs @@ -761,8 +761,12 @@ export function report(dir) { // file 95 historical command injections into the file-read balance -- exactly // the dilution the split exists to prevent. const isCommand = (e) => e.surface === 'command' || e.trigger === 'command'; - const commandInjections = allInjections.filter(isCommand); - const injections = allInjections.filter((e) => !isCommand(e)); + // Session-start delivery has no file anchor or downstream read join either. + // Report its cost independently so it cannot dilute the file-touch estimate. + const isSessionStart = (e) => e.surface === 'session-start'; + const sessionStartInjections = allInjections.filter(isSessionStart); + const commandInjections = allInjections.filter((e) => !isSessionStart(e) && isCommand(e)); + const injections = allInjections.filter((e) => !isSessionStart(e) && !isCommand(e)); const treated = injections.filter((e) => !e.holdout); const withheld = injections.filter((e) => e.holdout); @@ -840,6 +844,10 @@ export function report(dir) { return { injections: treated.length, + sessionStartInjections: sessionStartInjections.length, + sessionStartInjectedTokens: sessionStartInjections.reduce( + (sum, event) => sum + (event.deliveredTokens ?? event.tokens ?? 0), 0 + ), commandInjections: commandInjections.length, commandHoldouts: commandInjections.filter((e) => e.holdout).length, holdouts: withheld.length, @@ -1080,7 +1088,9 @@ function matchesFilters(event, filters) { if (!filters[field]) continue; let actual = event[field]; if (event.kind === 'handoff-run') { - actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + actual = field === 'taskId' + ? event.scenarioId + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); } @@ -1098,6 +1108,10 @@ function handoffArmMetrics(runs) { ); return { runs: runs.length, + delivery: proportionInterval( + runs.filter((run) => run.delivery?.delivered === true).length, + runs.length + ), correctness: proportionInterval(count('correct'), runs.length), firstPass: proportionInterval(count('firstPass'), runs.length), mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), @@ -1146,6 +1160,10 @@ const handoffKey = (run) => [ run.scenarioId || 'unknown', ].join('|'); +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; +const handoffMinimumPairs = () => + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + function handoffCohorts(runs) { const grouped = new Map(); for (const run of runs) { @@ -1153,7 +1171,7 @@ function handoffCohorts(runs) { if (!grouped.has(key)) grouped.set(key, []); grouped.get(key).push(run); } - const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + const minimumPairs = handoffMinimumPairs(); return [...grouped.entries()].map(([key, rows]) => { const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), @@ -1169,14 +1187,18 @@ function handoffCohorts(runs) { : null; const emptyRecurrence = arms.empty.mistakeExecuted.rate; const naturalRecurrence = arms.natural.mistakeExecuted.rate; - const relativeRecurrenceReduction = emptyRecurrence + const relativeRecurrenceReduction = emptyRecurrence !== null + && naturalRecurrence !== null + && emptyRecurrence !== 0 ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence : null; const controlsSafe = ['irrelevant', 'stale'].every((arm) => - arms[arm].correctness.rate === null - || arms.empty.correctness.rate === null - || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + arms[arm].correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 ); + const irrelevantSuppressed = arms.irrelevant.delivery.rate !== null + && arms.irrelevant.delivery.rate === 0; const gates = { minimumPairs: naturalVsEmpty.pairs >= minimumPairs, capture: captureRate !== null && captureRate >= 0.8, @@ -1184,9 +1206,11 @@ function handoffCohorts(runs) { relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, correctness: - (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + arms.natural.correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms.natural.correctness.rate >= arms.empty.correctness.rate - 0.1, preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, - negativeControls: controlsSafe, + negativeControls: controlsSafe && irrelevantSuppressed, }; const claimReady = Object.values(gates).every(Boolean); return { @@ -1209,8 +1233,6 @@ function handoffCohorts(runs) { }); } -const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; - function concurrencySummary(runs) { const natural = runs.filter((run) => run.arm === 'natural'); const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); @@ -1345,8 +1367,7 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, - handoffMinimumPairs: - Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), + handoffMinimumPairs: handoffMinimumPairs(), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/gemini/hooks/lib/adapter.mjs b/integrations/gemini/hooks/lib/adapter.mjs index c99d512cc..0e079eca7 100644 --- a/integrations/gemini/hooks/lib/adapter.mjs +++ b/integrations/gemini/hooks/lib/adapter.mjs @@ -437,7 +437,7 @@ export async function run(clientName, event) { // history even though this compact index never renders a diff. const graph = load(dir); const episode = episodeMeta({ client: clientName, raw }); - const rules = standingRules(dir, graph); + const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); const index = sessionIndex(dir, graph, { episode }); if (index) parts.push(index); diff --git a/integrations/gemini/hooks/lib/inject.mjs b/integrations/gemini/hooks/lib/inject.mjs index db1ecec4a..1ec5b4ce4 100644 --- a/integrations/gemini/hooks/lib/inject.mjs +++ b/integrations/gemini/hooks/lib/inject.mjs @@ -779,16 +779,24 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph, { episode = {} } = {}) { +export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); + const relevant = new Set(relevantFindingIds); + // SessionStart has no task signal. Fail closed for situational findings until + // a client supplies ids selected from an actual prompt/tool relevance pass. + // Universal human/pinned rules are delivered separately by `standingRules`. + if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. - const findings = [...graph.nodes.values()] + const allFindings = [...graph.nodes.values()] + .filter((n) => n.kind === 'finding'); + const findings = allFindings .filter((n) => - n.kind === 'finding' - && !n.retired + !n.retired && typeof n.claim === 'string' + && typeof n.key === 'string' + && relevant.has(n.key) // These are already rendered in full by `standingRules`; listing them a // second time spends tokens without adding information. && n.pinned !== true @@ -806,7 +814,7 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { // Staleness checks can touch disk, so bound the candidates BEFORE serving // them rather than checking an arbitrarily large graph on SessionStart. const candidate = [...selected, finding]; - const preview = renderSessionIndex(findings.length, candidate); + const preview = renderSessionIndex(allFindings.length, candidate); if (estimate(preview) > budget) break; selected.push(finding); } @@ -819,11 +827,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { if (!served.length) return null; // A stale marker is longer than the fresh preview used for candidate // selection. Trim from the lowest-ranked end until the ACTUAL message fits. - while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + while (served.length && estimate(renderSessionIndex(allFindings.length, served)) > budget) { served.pop(); } if (!served.length) return null; - const text = renderSessionIndex(findings.length, served); + const text = renderSessionIndex(allFindings.length, served); const spent = estimate(text); const findingIds = served.map((finding) => finding.key); @@ -857,9 +865,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { function renderSessionIndex(total, findings) { const lines = findings.map((finding) => { - const freshness = finding.stale + const freshness = finding.stale && finding.staleEvidence ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` - : ''; + : finding.stale + ? ' [possibly stale; verify before use]' + : ''; return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; }); return `# Project wiki (${total} findings, ${findings.length} listed) @@ -895,7 +905,7 @@ ${lines.join('\n')}`; * is the one that already happened here: an always-on block that grows until it * is wallpaper, and the model stops reading the thing it always sees. */ -export function standingRules(dir, graph, { budget = standingBudget() } = {}) { +export function standingRules(dir, graph, { budget = standingBudget(), episode = {} } = {}) { const rules = [...graph.nodes.values()].filter( (n) => n.kind === 'finding' && @@ -953,6 +963,7 @@ export function standingRules(dir, graph, { budget = standingBudget() } = {}) { // measurement shows no sign of it. const truncated = noticeFor(dropped); record(dir, { + ...episode, kind: 'standing', count: lines.length, dropped, diff --git a/integrations/gemini/hooks/lib/metrics.mjs b/integrations/gemini/hooks/lib/metrics.mjs index 3e4c4bb59..f6ab65780 100644 --- a/integrations/gemini/hooks/lib/metrics.mjs +++ b/integrations/gemini/hooks/lib/metrics.mjs @@ -761,8 +761,12 @@ export function report(dir) { // file 95 historical command injections into the file-read balance -- exactly // the dilution the split exists to prevent. const isCommand = (e) => e.surface === 'command' || e.trigger === 'command'; - const commandInjections = allInjections.filter(isCommand); - const injections = allInjections.filter((e) => !isCommand(e)); + // Session-start delivery has no file anchor or downstream read join either. + // Report its cost independently so it cannot dilute the file-touch estimate. + const isSessionStart = (e) => e.surface === 'session-start'; + const sessionStartInjections = allInjections.filter(isSessionStart); + const commandInjections = allInjections.filter((e) => !isSessionStart(e) && isCommand(e)); + const injections = allInjections.filter((e) => !isSessionStart(e) && !isCommand(e)); const treated = injections.filter((e) => !e.holdout); const withheld = injections.filter((e) => e.holdout); @@ -840,6 +844,10 @@ export function report(dir) { return { injections: treated.length, + sessionStartInjections: sessionStartInjections.length, + sessionStartInjectedTokens: sessionStartInjections.reduce( + (sum, event) => sum + (event.deliveredTokens ?? event.tokens ?? 0), 0 + ), commandInjections: commandInjections.length, commandHoldouts: commandInjections.filter((e) => e.holdout).length, holdouts: withheld.length, @@ -1080,7 +1088,9 @@ function matchesFilters(event, filters) { if (!filters[field]) continue; let actual = event[field]; if (event.kind === 'handoff-run') { - actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + actual = field === 'taskId' + ? event.scenarioId + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); } @@ -1098,6 +1108,10 @@ function handoffArmMetrics(runs) { ); return { runs: runs.length, + delivery: proportionInterval( + runs.filter((run) => run.delivery?.delivered === true).length, + runs.length + ), correctness: proportionInterval(count('correct'), runs.length), firstPass: proportionInterval(count('firstPass'), runs.length), mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), @@ -1146,6 +1160,10 @@ const handoffKey = (run) => [ run.scenarioId || 'unknown', ].join('|'); +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; +const handoffMinimumPairs = () => + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + function handoffCohorts(runs) { const grouped = new Map(); for (const run of runs) { @@ -1153,7 +1171,7 @@ function handoffCohorts(runs) { if (!grouped.has(key)) grouped.set(key, []); grouped.get(key).push(run); } - const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + const minimumPairs = handoffMinimumPairs(); return [...grouped.entries()].map(([key, rows]) => { const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), @@ -1169,14 +1187,18 @@ function handoffCohorts(runs) { : null; const emptyRecurrence = arms.empty.mistakeExecuted.rate; const naturalRecurrence = arms.natural.mistakeExecuted.rate; - const relativeRecurrenceReduction = emptyRecurrence + const relativeRecurrenceReduction = emptyRecurrence !== null + && naturalRecurrence !== null + && emptyRecurrence !== 0 ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence : null; const controlsSafe = ['irrelevant', 'stale'].every((arm) => - arms[arm].correctness.rate === null - || arms.empty.correctness.rate === null - || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + arms[arm].correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 ); + const irrelevantSuppressed = arms.irrelevant.delivery.rate !== null + && arms.irrelevant.delivery.rate === 0; const gates = { minimumPairs: naturalVsEmpty.pairs >= minimumPairs, capture: captureRate !== null && captureRate >= 0.8, @@ -1184,9 +1206,11 @@ function handoffCohorts(runs) { relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, correctness: - (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + arms.natural.correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms.natural.correctness.rate >= arms.empty.correctness.rate - 0.1, preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, - negativeControls: controlsSafe, + negativeControls: controlsSafe && irrelevantSuppressed, }; const claimReady = Object.values(gates).every(Boolean); return { @@ -1209,8 +1233,6 @@ function handoffCohorts(runs) { }); } -const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; - function concurrencySummary(runs) { const natural = runs.filter((run) => run.arm === 'natural'); const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); @@ -1345,8 +1367,7 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, - handoffMinimumPairs: - Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), + handoffMinimumPairs: handoffMinimumPairs(), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/kilo/hooks/lib/adapter.mjs b/integrations/kilo/hooks/lib/adapter.mjs index c99d512cc..0e079eca7 100644 --- a/integrations/kilo/hooks/lib/adapter.mjs +++ b/integrations/kilo/hooks/lib/adapter.mjs @@ -437,7 +437,7 @@ export async function run(clientName, event) { // history even though this compact index never renders a diff. const graph = load(dir); const episode = episodeMeta({ client: clientName, raw }); - const rules = standingRules(dir, graph); + const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); const index = sessionIndex(dir, graph, { episode }); if (index) parts.push(index); diff --git a/integrations/kilo/hooks/lib/inject.mjs b/integrations/kilo/hooks/lib/inject.mjs index db1ecec4a..1ec5b4ce4 100644 --- a/integrations/kilo/hooks/lib/inject.mjs +++ b/integrations/kilo/hooks/lib/inject.mjs @@ -779,16 +779,24 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph, { episode = {} } = {}) { +export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); + const relevant = new Set(relevantFindingIds); + // SessionStart has no task signal. Fail closed for situational findings until + // a client supplies ids selected from an actual prompt/tool relevance pass. + // Universal human/pinned rules are delivered separately by `standingRules`. + if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. - const findings = [...graph.nodes.values()] + const allFindings = [...graph.nodes.values()] + .filter((n) => n.kind === 'finding'); + const findings = allFindings .filter((n) => - n.kind === 'finding' - && !n.retired + !n.retired && typeof n.claim === 'string' + && typeof n.key === 'string' + && relevant.has(n.key) // These are already rendered in full by `standingRules`; listing them a // second time spends tokens without adding information. && n.pinned !== true @@ -806,7 +814,7 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { // Staleness checks can touch disk, so bound the candidates BEFORE serving // them rather than checking an arbitrarily large graph on SessionStart. const candidate = [...selected, finding]; - const preview = renderSessionIndex(findings.length, candidate); + const preview = renderSessionIndex(allFindings.length, candidate); if (estimate(preview) > budget) break; selected.push(finding); } @@ -819,11 +827,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { if (!served.length) return null; // A stale marker is longer than the fresh preview used for candidate // selection. Trim from the lowest-ranked end until the ACTUAL message fits. - while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + while (served.length && estimate(renderSessionIndex(allFindings.length, served)) > budget) { served.pop(); } if (!served.length) return null; - const text = renderSessionIndex(findings.length, served); + const text = renderSessionIndex(allFindings.length, served); const spent = estimate(text); const findingIds = served.map((finding) => finding.key); @@ -857,9 +865,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { function renderSessionIndex(total, findings) { const lines = findings.map((finding) => { - const freshness = finding.stale + const freshness = finding.stale && finding.staleEvidence ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` - : ''; + : finding.stale + ? ' [possibly stale; verify before use]' + : ''; return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; }); return `# Project wiki (${total} findings, ${findings.length} listed) @@ -895,7 +905,7 @@ ${lines.join('\n')}`; * is the one that already happened here: an always-on block that grows until it * is wallpaper, and the model stops reading the thing it always sees. */ -export function standingRules(dir, graph, { budget = standingBudget() } = {}) { +export function standingRules(dir, graph, { budget = standingBudget(), episode = {} } = {}) { const rules = [...graph.nodes.values()].filter( (n) => n.kind === 'finding' && @@ -953,6 +963,7 @@ export function standingRules(dir, graph, { budget = standingBudget() } = {}) { // measurement shows no sign of it. const truncated = noticeFor(dropped); record(dir, { + ...episode, kind: 'standing', count: lines.length, dropped, diff --git a/integrations/kilo/hooks/lib/metrics.mjs b/integrations/kilo/hooks/lib/metrics.mjs index 3e4c4bb59..f6ab65780 100644 --- a/integrations/kilo/hooks/lib/metrics.mjs +++ b/integrations/kilo/hooks/lib/metrics.mjs @@ -761,8 +761,12 @@ export function report(dir) { // file 95 historical command injections into the file-read balance -- exactly // the dilution the split exists to prevent. const isCommand = (e) => e.surface === 'command' || e.trigger === 'command'; - const commandInjections = allInjections.filter(isCommand); - const injections = allInjections.filter((e) => !isCommand(e)); + // Session-start delivery has no file anchor or downstream read join either. + // Report its cost independently so it cannot dilute the file-touch estimate. + const isSessionStart = (e) => e.surface === 'session-start'; + const sessionStartInjections = allInjections.filter(isSessionStart); + const commandInjections = allInjections.filter((e) => !isSessionStart(e) && isCommand(e)); + const injections = allInjections.filter((e) => !isSessionStart(e) && !isCommand(e)); const treated = injections.filter((e) => !e.holdout); const withheld = injections.filter((e) => e.holdout); @@ -840,6 +844,10 @@ export function report(dir) { return { injections: treated.length, + sessionStartInjections: sessionStartInjections.length, + sessionStartInjectedTokens: sessionStartInjections.reduce( + (sum, event) => sum + (event.deliveredTokens ?? event.tokens ?? 0), 0 + ), commandInjections: commandInjections.length, commandHoldouts: commandInjections.filter((e) => e.holdout).length, holdouts: withheld.length, @@ -1080,7 +1088,9 @@ function matchesFilters(event, filters) { if (!filters[field]) continue; let actual = event[field]; if (event.kind === 'handoff-run') { - actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + actual = field === 'taskId' + ? event.scenarioId + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); } @@ -1098,6 +1108,10 @@ function handoffArmMetrics(runs) { ); return { runs: runs.length, + delivery: proportionInterval( + runs.filter((run) => run.delivery?.delivered === true).length, + runs.length + ), correctness: proportionInterval(count('correct'), runs.length), firstPass: proportionInterval(count('firstPass'), runs.length), mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), @@ -1146,6 +1160,10 @@ const handoffKey = (run) => [ run.scenarioId || 'unknown', ].join('|'); +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; +const handoffMinimumPairs = () => + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + function handoffCohorts(runs) { const grouped = new Map(); for (const run of runs) { @@ -1153,7 +1171,7 @@ function handoffCohorts(runs) { if (!grouped.has(key)) grouped.set(key, []); grouped.get(key).push(run); } - const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + const minimumPairs = handoffMinimumPairs(); return [...grouped.entries()].map(([key, rows]) => { const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), @@ -1169,14 +1187,18 @@ function handoffCohorts(runs) { : null; const emptyRecurrence = arms.empty.mistakeExecuted.rate; const naturalRecurrence = arms.natural.mistakeExecuted.rate; - const relativeRecurrenceReduction = emptyRecurrence + const relativeRecurrenceReduction = emptyRecurrence !== null + && naturalRecurrence !== null + && emptyRecurrence !== 0 ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence : null; const controlsSafe = ['irrelevant', 'stale'].every((arm) => - arms[arm].correctness.rate === null - || arms.empty.correctness.rate === null - || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + arms[arm].correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 ); + const irrelevantSuppressed = arms.irrelevant.delivery.rate !== null + && arms.irrelevant.delivery.rate === 0; const gates = { minimumPairs: naturalVsEmpty.pairs >= minimumPairs, capture: captureRate !== null && captureRate >= 0.8, @@ -1184,9 +1206,11 @@ function handoffCohorts(runs) { relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, correctness: - (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + arms.natural.correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms.natural.correctness.rate >= arms.empty.correctness.rate - 0.1, preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, - negativeControls: controlsSafe, + negativeControls: controlsSafe && irrelevantSuppressed, }; const claimReady = Object.values(gates).every(Boolean); return { @@ -1209,8 +1233,6 @@ function handoffCohorts(runs) { }); } -const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; - function concurrencySummary(runs) { const natural = runs.filter((run) => run.arm === 'natural'); const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); @@ -1345,8 +1367,7 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, - handoffMinimumPairs: - Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), + handoffMinimumPairs: handoffMinimumPairs(), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/opencode/hooks/lib/adapter.mjs b/integrations/opencode/hooks/lib/adapter.mjs index c99d512cc..0e079eca7 100644 --- a/integrations/opencode/hooks/lib/adapter.mjs +++ b/integrations/opencode/hooks/lib/adapter.mjs @@ -437,7 +437,7 @@ export async function run(clientName, event) { // history even though this compact index never renders a diff. const graph = load(dir); const episode = episodeMeta({ client: clientName, raw }); - const rules = standingRules(dir, graph); + const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); const index = sessionIndex(dir, graph, { episode }); if (index) parts.push(index); diff --git a/integrations/opencode/hooks/lib/inject.mjs b/integrations/opencode/hooks/lib/inject.mjs index db1ecec4a..1ec5b4ce4 100644 --- a/integrations/opencode/hooks/lib/inject.mjs +++ b/integrations/opencode/hooks/lib/inject.mjs @@ -779,16 +779,24 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph, { episode = {} } = {}) { +export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); + const relevant = new Set(relevantFindingIds); + // SessionStart has no task signal. Fail closed for situational findings until + // a client supplies ids selected from an actual prompt/tool relevance pass. + // Universal human/pinned rules are delivered separately by `standingRules`. + if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. - const findings = [...graph.nodes.values()] + const allFindings = [...graph.nodes.values()] + .filter((n) => n.kind === 'finding'); + const findings = allFindings .filter((n) => - n.kind === 'finding' - && !n.retired + !n.retired && typeof n.claim === 'string' + && typeof n.key === 'string' + && relevant.has(n.key) // These are already rendered in full by `standingRules`; listing them a // second time spends tokens without adding information. && n.pinned !== true @@ -806,7 +814,7 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { // Staleness checks can touch disk, so bound the candidates BEFORE serving // them rather than checking an arbitrarily large graph on SessionStart. const candidate = [...selected, finding]; - const preview = renderSessionIndex(findings.length, candidate); + const preview = renderSessionIndex(allFindings.length, candidate); if (estimate(preview) > budget) break; selected.push(finding); } @@ -819,11 +827,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { if (!served.length) return null; // A stale marker is longer than the fresh preview used for candidate // selection. Trim from the lowest-ranked end until the ACTUAL message fits. - while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + while (served.length && estimate(renderSessionIndex(allFindings.length, served)) > budget) { served.pop(); } if (!served.length) return null; - const text = renderSessionIndex(findings.length, served); + const text = renderSessionIndex(allFindings.length, served); const spent = estimate(text); const findingIds = served.map((finding) => finding.key); @@ -857,9 +865,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { function renderSessionIndex(total, findings) { const lines = findings.map((finding) => { - const freshness = finding.stale + const freshness = finding.stale && finding.staleEvidence ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` - : ''; + : finding.stale + ? ' [possibly stale; verify before use]' + : ''; return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; }); return `# Project wiki (${total} findings, ${findings.length} listed) @@ -895,7 +905,7 @@ ${lines.join('\n')}`; * is the one that already happened here: an always-on block that grows until it * is wallpaper, and the model stops reading the thing it always sees. */ -export function standingRules(dir, graph, { budget = standingBudget() } = {}) { +export function standingRules(dir, graph, { budget = standingBudget(), episode = {} } = {}) { const rules = [...graph.nodes.values()].filter( (n) => n.kind === 'finding' && @@ -953,6 +963,7 @@ export function standingRules(dir, graph, { budget = standingBudget() } = {}) { // measurement shows no sign of it. const truncated = noticeFor(dropped); record(dir, { + ...episode, kind: 'standing', count: lines.length, dropped, diff --git a/integrations/opencode/hooks/lib/metrics.mjs b/integrations/opencode/hooks/lib/metrics.mjs index 3e4c4bb59..f6ab65780 100644 --- a/integrations/opencode/hooks/lib/metrics.mjs +++ b/integrations/opencode/hooks/lib/metrics.mjs @@ -761,8 +761,12 @@ export function report(dir) { // file 95 historical command injections into the file-read balance -- exactly // the dilution the split exists to prevent. const isCommand = (e) => e.surface === 'command' || e.trigger === 'command'; - const commandInjections = allInjections.filter(isCommand); - const injections = allInjections.filter((e) => !isCommand(e)); + // Session-start delivery has no file anchor or downstream read join either. + // Report its cost independently so it cannot dilute the file-touch estimate. + const isSessionStart = (e) => e.surface === 'session-start'; + const sessionStartInjections = allInjections.filter(isSessionStart); + const commandInjections = allInjections.filter((e) => !isSessionStart(e) && isCommand(e)); + const injections = allInjections.filter((e) => !isSessionStart(e) && !isCommand(e)); const treated = injections.filter((e) => !e.holdout); const withheld = injections.filter((e) => e.holdout); @@ -840,6 +844,10 @@ export function report(dir) { return { injections: treated.length, + sessionStartInjections: sessionStartInjections.length, + sessionStartInjectedTokens: sessionStartInjections.reduce( + (sum, event) => sum + (event.deliveredTokens ?? event.tokens ?? 0), 0 + ), commandInjections: commandInjections.length, commandHoldouts: commandInjections.filter((e) => e.holdout).length, holdouts: withheld.length, @@ -1080,7 +1088,9 @@ function matchesFilters(event, filters) { if (!filters[field]) continue; let actual = event[field]; if (event.kind === 'handoff-run') { - actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + actual = field === 'taskId' + ? event.scenarioId + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); } @@ -1098,6 +1108,10 @@ function handoffArmMetrics(runs) { ); return { runs: runs.length, + delivery: proportionInterval( + runs.filter((run) => run.delivery?.delivered === true).length, + runs.length + ), correctness: proportionInterval(count('correct'), runs.length), firstPass: proportionInterval(count('firstPass'), runs.length), mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), @@ -1146,6 +1160,10 @@ const handoffKey = (run) => [ run.scenarioId || 'unknown', ].join('|'); +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; +const handoffMinimumPairs = () => + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + function handoffCohorts(runs) { const grouped = new Map(); for (const run of runs) { @@ -1153,7 +1171,7 @@ function handoffCohorts(runs) { if (!grouped.has(key)) grouped.set(key, []); grouped.get(key).push(run); } - const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + const minimumPairs = handoffMinimumPairs(); return [...grouped.entries()].map(([key, rows]) => { const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), @@ -1169,14 +1187,18 @@ function handoffCohorts(runs) { : null; const emptyRecurrence = arms.empty.mistakeExecuted.rate; const naturalRecurrence = arms.natural.mistakeExecuted.rate; - const relativeRecurrenceReduction = emptyRecurrence + const relativeRecurrenceReduction = emptyRecurrence !== null + && naturalRecurrence !== null + && emptyRecurrence !== 0 ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence : null; const controlsSafe = ['irrelevant', 'stale'].every((arm) => - arms[arm].correctness.rate === null - || arms.empty.correctness.rate === null - || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + arms[arm].correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 ); + const irrelevantSuppressed = arms.irrelevant.delivery.rate !== null + && arms.irrelevant.delivery.rate === 0; const gates = { minimumPairs: naturalVsEmpty.pairs >= minimumPairs, capture: captureRate !== null && captureRate >= 0.8, @@ -1184,9 +1206,11 @@ function handoffCohorts(runs) { relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, correctness: - (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + arms.natural.correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms.natural.correctness.rate >= arms.empty.correctness.rate - 0.1, preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, - negativeControls: controlsSafe, + negativeControls: controlsSafe && irrelevantSuppressed, }; const claimReady = Object.values(gates).every(Boolean); return { @@ -1209,8 +1233,6 @@ function handoffCohorts(runs) { }); } -const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; - function concurrencySummary(runs) { const natural = runs.filter((run) => run.arm === 'natural'); const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); @@ -1345,8 +1367,7 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, - handoffMinimumPairs: - Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), + handoffMinimumPairs: handoffMinimumPairs(), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/qwen/hooks/lib/adapter.mjs b/integrations/qwen/hooks/lib/adapter.mjs index c99d512cc..0e079eca7 100644 --- a/integrations/qwen/hooks/lib/adapter.mjs +++ b/integrations/qwen/hooks/lib/adapter.mjs @@ -437,7 +437,7 @@ export async function run(clientName, event) { // history even though this compact index never renders a diff. const graph = load(dir); const episode = episodeMeta({ client: clientName, raw }); - const rules = standingRules(dir, graph); + const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); const index = sessionIndex(dir, graph, { episode }); if (index) parts.push(index); diff --git a/integrations/qwen/hooks/lib/inject.mjs b/integrations/qwen/hooks/lib/inject.mjs index db1ecec4a..1ec5b4ce4 100644 --- a/integrations/qwen/hooks/lib/inject.mjs +++ b/integrations/qwen/hooks/lib/inject.mjs @@ -779,16 +779,24 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph, { episode = {} } = {}) { +export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); + const relevant = new Set(relevantFindingIds); + // SessionStart has no task signal. Fail closed for situational findings until + // a client supplies ids selected from an actual prompt/tool relevance pass. + // Universal human/pinned rules are delivered separately by `standingRules`. + if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. - const findings = [...graph.nodes.values()] + const allFindings = [...graph.nodes.values()] + .filter((n) => n.kind === 'finding'); + const findings = allFindings .filter((n) => - n.kind === 'finding' - && !n.retired + !n.retired && typeof n.claim === 'string' + && typeof n.key === 'string' + && relevant.has(n.key) // These are already rendered in full by `standingRules`; listing them a // second time spends tokens without adding information. && n.pinned !== true @@ -806,7 +814,7 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { // Staleness checks can touch disk, so bound the candidates BEFORE serving // them rather than checking an arbitrarily large graph on SessionStart. const candidate = [...selected, finding]; - const preview = renderSessionIndex(findings.length, candidate); + const preview = renderSessionIndex(allFindings.length, candidate); if (estimate(preview) > budget) break; selected.push(finding); } @@ -819,11 +827,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { if (!served.length) return null; // A stale marker is longer than the fresh preview used for candidate // selection. Trim from the lowest-ranked end until the ACTUAL message fits. - while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + while (served.length && estimate(renderSessionIndex(allFindings.length, served)) > budget) { served.pop(); } if (!served.length) return null; - const text = renderSessionIndex(findings.length, served); + const text = renderSessionIndex(allFindings.length, served); const spent = estimate(text); const findingIds = served.map((finding) => finding.key); @@ -857,9 +865,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { function renderSessionIndex(total, findings) { const lines = findings.map((finding) => { - const freshness = finding.stale + const freshness = finding.stale && finding.staleEvidence ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` - : ''; + : finding.stale + ? ' [possibly stale; verify before use]' + : ''; return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; }); return `# Project wiki (${total} findings, ${findings.length} listed) @@ -895,7 +905,7 @@ ${lines.join('\n')}`; * is the one that already happened here: an always-on block that grows until it * is wallpaper, and the model stops reading the thing it always sees. */ -export function standingRules(dir, graph, { budget = standingBudget() } = {}) { +export function standingRules(dir, graph, { budget = standingBudget(), episode = {} } = {}) { const rules = [...graph.nodes.values()].filter( (n) => n.kind === 'finding' && @@ -953,6 +963,7 @@ export function standingRules(dir, graph, { budget = standingBudget() } = {}) { // measurement shows no sign of it. const truncated = noticeFor(dropped); record(dir, { + ...episode, kind: 'standing', count: lines.length, dropped, diff --git a/integrations/qwen/hooks/lib/metrics.mjs b/integrations/qwen/hooks/lib/metrics.mjs index 3e4c4bb59..f6ab65780 100644 --- a/integrations/qwen/hooks/lib/metrics.mjs +++ b/integrations/qwen/hooks/lib/metrics.mjs @@ -761,8 +761,12 @@ export function report(dir) { // file 95 historical command injections into the file-read balance -- exactly // the dilution the split exists to prevent. const isCommand = (e) => e.surface === 'command' || e.trigger === 'command'; - const commandInjections = allInjections.filter(isCommand); - const injections = allInjections.filter((e) => !isCommand(e)); + // Session-start delivery has no file anchor or downstream read join either. + // Report its cost independently so it cannot dilute the file-touch estimate. + const isSessionStart = (e) => e.surface === 'session-start'; + const sessionStartInjections = allInjections.filter(isSessionStart); + const commandInjections = allInjections.filter((e) => !isSessionStart(e) && isCommand(e)); + const injections = allInjections.filter((e) => !isSessionStart(e) && !isCommand(e)); const treated = injections.filter((e) => !e.holdout); const withheld = injections.filter((e) => e.holdout); @@ -840,6 +844,10 @@ export function report(dir) { return { injections: treated.length, + sessionStartInjections: sessionStartInjections.length, + sessionStartInjectedTokens: sessionStartInjections.reduce( + (sum, event) => sum + (event.deliveredTokens ?? event.tokens ?? 0), 0 + ), commandInjections: commandInjections.length, commandHoldouts: commandInjections.filter((e) => e.holdout).length, holdouts: withheld.length, @@ -1080,7 +1088,9 @@ function matchesFilters(event, filters) { if (!filters[field]) continue; let actual = event[field]; if (event.kind === 'handoff-run') { - actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + actual = field === 'taskId' + ? event.scenarioId + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); } @@ -1098,6 +1108,10 @@ function handoffArmMetrics(runs) { ); return { runs: runs.length, + delivery: proportionInterval( + runs.filter((run) => run.delivery?.delivered === true).length, + runs.length + ), correctness: proportionInterval(count('correct'), runs.length), firstPass: proportionInterval(count('firstPass'), runs.length), mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), @@ -1146,6 +1160,10 @@ const handoffKey = (run) => [ run.scenarioId || 'unknown', ].join('|'); +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; +const handoffMinimumPairs = () => + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + function handoffCohorts(runs) { const grouped = new Map(); for (const run of runs) { @@ -1153,7 +1171,7 @@ function handoffCohorts(runs) { if (!grouped.has(key)) grouped.set(key, []); grouped.get(key).push(run); } - const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + const minimumPairs = handoffMinimumPairs(); return [...grouped.entries()].map(([key, rows]) => { const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), @@ -1169,14 +1187,18 @@ function handoffCohorts(runs) { : null; const emptyRecurrence = arms.empty.mistakeExecuted.rate; const naturalRecurrence = arms.natural.mistakeExecuted.rate; - const relativeRecurrenceReduction = emptyRecurrence + const relativeRecurrenceReduction = emptyRecurrence !== null + && naturalRecurrence !== null + && emptyRecurrence !== 0 ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence : null; const controlsSafe = ['irrelevant', 'stale'].every((arm) => - arms[arm].correctness.rate === null - || arms.empty.correctness.rate === null - || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + arms[arm].correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 ); + const irrelevantSuppressed = arms.irrelevant.delivery.rate !== null + && arms.irrelevant.delivery.rate === 0; const gates = { minimumPairs: naturalVsEmpty.pairs >= minimumPairs, capture: captureRate !== null && captureRate >= 0.8, @@ -1184,9 +1206,11 @@ function handoffCohorts(runs) { relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, correctness: - (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + arms.natural.correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms.natural.correctness.rate >= arms.empty.correctness.rate - 0.1, preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, - negativeControls: controlsSafe, + negativeControls: controlsSafe && irrelevantSuppressed, }; const claimReady = Object.values(gates).every(Boolean); return { @@ -1209,8 +1233,6 @@ function handoffCohorts(runs) { }); } -const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; - function concurrencySummary(runs) { const natural = runs.filter((run) => run.arm === 'natural'); const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); @@ -1345,8 +1367,7 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, - handoffMinimumPairs: - Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), + handoffMinimumPairs: handoffMinimumPairs(), deterministicChecksAreCausalProof: false, }, }; diff --git a/integrations/windsurf/hooks/lib/adapter.mjs b/integrations/windsurf/hooks/lib/adapter.mjs index c99d512cc..0e079eca7 100644 --- a/integrations/windsurf/hooks/lib/adapter.mjs +++ b/integrations/windsurf/hooks/lib/adapter.mjs @@ -437,7 +437,7 @@ export async function run(clientName, event) { // history even though this compact index never renders a diff. const graph = load(dir); const episode = episodeMeta({ client: clientName, raw }); - const rules = standingRules(dir, graph); + const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); const index = sessionIndex(dir, graph, { episode }); if (index) parts.push(index); diff --git a/integrations/windsurf/hooks/lib/inject.mjs b/integrations/windsurf/hooks/lib/inject.mjs index db1ecec4a..1ec5b4ce4 100644 --- a/integrations/windsurf/hooks/lib/inject.mjs +++ b/integrations/windsurf/hooks/lib/inject.mjs @@ -779,16 +779,24 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph, { episode = {} } = {}) { +export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); + const relevant = new Set(relevantFindingIds); + // SessionStart has no task signal. Fail closed for situational findings until + // a client supplies ids selected from an actual prompt/tool relevance pass. + // Universal human/pinned rules are delivered separately by `standingRules`. + if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. - const findings = [...graph.nodes.values()] + const allFindings = [...graph.nodes.values()] + .filter((n) => n.kind === 'finding'); + const findings = allFindings .filter((n) => - n.kind === 'finding' - && !n.retired + !n.retired && typeof n.claim === 'string' + && typeof n.key === 'string' + && relevant.has(n.key) // These are already rendered in full by `standingRules`; listing them a // second time spends tokens without adding information. && n.pinned !== true @@ -806,7 +814,7 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { // Staleness checks can touch disk, so bound the candidates BEFORE serving // them rather than checking an arbitrarily large graph on SessionStart. const candidate = [...selected, finding]; - const preview = renderSessionIndex(findings.length, candidate); + const preview = renderSessionIndex(allFindings.length, candidate); if (estimate(preview) > budget) break; selected.push(finding); } @@ -819,11 +827,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { if (!served.length) return null; // A stale marker is longer than the fresh preview used for candidate // selection. Trim from the lowest-ranked end until the ACTUAL message fits. - while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + while (served.length && estimate(renderSessionIndex(allFindings.length, served)) > budget) { served.pop(); } if (!served.length) return null; - const text = renderSessionIndex(findings.length, served); + const text = renderSessionIndex(allFindings.length, served); const spent = estimate(text); const findingIds = served.map((finding) => finding.key); @@ -857,9 +865,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { function renderSessionIndex(total, findings) { const lines = findings.map((finding) => { - const freshness = finding.stale + const freshness = finding.stale && finding.staleEvidence ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` - : ''; + : finding.stale + ? ' [possibly stale; verify before use]' + : ''; return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; }); return `# Project wiki (${total} findings, ${findings.length} listed) @@ -895,7 +905,7 @@ ${lines.join('\n')}`; * is the one that already happened here: an always-on block that grows until it * is wallpaper, and the model stops reading the thing it always sees. */ -export function standingRules(dir, graph, { budget = standingBudget() } = {}) { +export function standingRules(dir, graph, { budget = standingBudget(), episode = {} } = {}) { const rules = [...graph.nodes.values()].filter( (n) => n.kind === 'finding' && @@ -953,6 +963,7 @@ export function standingRules(dir, graph, { budget = standingBudget() } = {}) { // measurement shows no sign of it. const truncated = noticeFor(dropped); record(dir, { + ...episode, kind: 'standing', count: lines.length, dropped, diff --git a/integrations/windsurf/hooks/lib/metrics.mjs b/integrations/windsurf/hooks/lib/metrics.mjs index 3e4c4bb59..f6ab65780 100644 --- a/integrations/windsurf/hooks/lib/metrics.mjs +++ b/integrations/windsurf/hooks/lib/metrics.mjs @@ -761,8 +761,12 @@ export function report(dir) { // file 95 historical command injections into the file-read balance -- exactly // the dilution the split exists to prevent. const isCommand = (e) => e.surface === 'command' || e.trigger === 'command'; - const commandInjections = allInjections.filter(isCommand); - const injections = allInjections.filter((e) => !isCommand(e)); + // Session-start delivery has no file anchor or downstream read join either. + // Report its cost independently so it cannot dilute the file-touch estimate. + const isSessionStart = (e) => e.surface === 'session-start'; + const sessionStartInjections = allInjections.filter(isSessionStart); + const commandInjections = allInjections.filter((e) => !isSessionStart(e) && isCommand(e)); + const injections = allInjections.filter((e) => !isSessionStart(e) && !isCommand(e)); const treated = injections.filter((e) => !e.holdout); const withheld = injections.filter((e) => e.holdout); @@ -840,6 +844,10 @@ export function report(dir) { return { injections: treated.length, + sessionStartInjections: sessionStartInjections.length, + sessionStartInjectedTokens: sessionStartInjections.reduce( + (sum, event) => sum + (event.deliveredTokens ?? event.tokens ?? 0), 0 + ), commandInjections: commandInjections.length, commandHoldouts: commandInjections.filter((e) => e.holdout).length, holdouts: withheld.length, @@ -1080,7 +1088,9 @@ function matchesFilters(event, filters) { if (!filters[field]) continue; let actual = event[field]; if (event.kind === 'handoff-run') { - actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + actual = field === 'taskId' + ? event.scenarioId + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); } @@ -1098,6 +1108,10 @@ function handoffArmMetrics(runs) { ); return { runs: runs.length, + delivery: proportionInterval( + runs.filter((run) => run.delivery?.delivered === true).length, + runs.length + ), correctness: proportionInterval(count('correct'), runs.length), firstPass: proportionInterval(count('firstPass'), runs.length), mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), @@ -1146,6 +1160,10 @@ const handoffKey = (run) => [ run.scenarioId || 'unknown', ].join('|'); +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; +const handoffMinimumPairs = () => + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + function handoffCohorts(runs) { const grouped = new Map(); for (const run of runs) { @@ -1153,7 +1171,7 @@ function handoffCohorts(runs) { if (!grouped.has(key)) grouped.set(key, []); grouped.get(key).push(run); } - const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + const minimumPairs = handoffMinimumPairs(); return [...grouped.entries()].map(([key, rows]) => { const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), @@ -1169,14 +1187,18 @@ function handoffCohorts(runs) { : null; const emptyRecurrence = arms.empty.mistakeExecuted.rate; const naturalRecurrence = arms.natural.mistakeExecuted.rate; - const relativeRecurrenceReduction = emptyRecurrence + const relativeRecurrenceReduction = emptyRecurrence !== null + && naturalRecurrence !== null + && emptyRecurrence !== 0 ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence : null; const controlsSafe = ['irrelevant', 'stale'].every((arm) => - arms[arm].correctness.rate === null - || arms.empty.correctness.rate === null - || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + arms[arm].correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 ); + const irrelevantSuppressed = arms.irrelevant.delivery.rate !== null + && arms.irrelevant.delivery.rate === 0; const gates = { minimumPairs: naturalVsEmpty.pairs >= minimumPairs, capture: captureRate !== null && captureRate >= 0.8, @@ -1184,9 +1206,11 @@ function handoffCohorts(runs) { relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, correctness: - (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + arms.natural.correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms.natural.correctness.rate >= arms.empty.correctness.rate - 0.1, preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, - negativeControls: controlsSafe, + negativeControls: controlsSafe && irrelevantSuppressed, }; const claimReady = Object.values(gates).every(Boolean); return { @@ -1209,8 +1233,6 @@ function handoffCohorts(runs) { }); } -const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; - function concurrencySummary(runs) { const natural = runs.filter((run) => run.arm === 'natural'); const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); @@ -1345,8 +1367,7 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, - handoffMinimumPairs: - Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), + handoffMinimumPairs: handoffMinimumPairs(), deterministicChecksAreCausalProof: false, }, }; diff --git a/plugin/hooks/lib/adapter.mjs b/plugin/hooks/lib/adapter.mjs index c99d512cc..0e079eca7 100644 --- a/plugin/hooks/lib/adapter.mjs +++ b/plugin/hooks/lib/adapter.mjs @@ -437,7 +437,7 @@ export async function run(clientName, event) { // history even though this compact index never renders a diff. const graph = load(dir); const episode = episodeMeta({ client: clientName, raw }); - const rules = standingRules(dir, graph); + const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); const index = sessionIndex(dir, graph, { episode }); if (index) parts.push(index); diff --git a/plugin/hooks/lib/inject.mjs b/plugin/hooks/lib/inject.mjs index db1ecec4a..1ec5b4ce4 100644 --- a/plugin/hooks/lib/inject.mjs +++ b/plugin/hooks/lib/inject.mjs @@ -779,16 +779,24 @@ export function forSharedCommand( * graph that demonstrably gets queried grows its allowance while a noisy one * shrinks toward the floor. See metrics.indexBudget. */ -export function sessionIndex(dir, graph, { episode = {} } = {}) { +export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); + const relevant = new Set(relevantFindingIds); + // SessionStart has no task signal. Fail closed for situational findings until + // a client supplies ids selected from an actual prompt/tool relevance pass. + // Universal human/pinned rules are delivered separately by `standingRules`. + if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read // path, so listing them here would advertise claims a human has explicitly // withdrawn -- and the index is the first thing the model reads. - const findings = [...graph.nodes.values()] + const allFindings = [...graph.nodes.values()] + .filter((n) => n.kind === 'finding'); + const findings = allFindings .filter((n) => - n.kind === 'finding' - && !n.retired + !n.retired && typeof n.claim === 'string' + && typeof n.key === 'string' + && relevant.has(n.key) // These are already rendered in full by `standingRules`; listing them a // second time spends tokens without adding information. && n.pinned !== true @@ -806,7 +814,7 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { // Staleness checks can touch disk, so bound the candidates BEFORE serving // them rather than checking an arbitrarily large graph on SessionStart. const candidate = [...selected, finding]; - const preview = renderSessionIndex(findings.length, candidate); + const preview = renderSessionIndex(allFindings.length, candidate); if (estimate(preview) > budget) break; selected.push(finding); } @@ -819,11 +827,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { if (!served.length) return null; // A stale marker is longer than the fresh preview used for candidate // selection. Trim from the lowest-ranked end until the ACTUAL message fits. - while (served.length && estimate(renderSessionIndex(findings.length, served)) > budget) { + while (served.length && estimate(renderSessionIndex(allFindings.length, served)) > budget) { served.pop(); } if (!served.length) return null; - const text = renderSessionIndex(findings.length, served); + const text = renderSessionIndex(allFindings.length, served); const spent = estimate(text); const findingIds = served.map((finding) => finding.key); @@ -857,9 +865,11 @@ export function sessionIndex(dir, graph, { episode = {} } = {}) { function renderSessionIndex(total, findings) { const lines = findings.map((finding) => { - const freshness = finding.stale + const freshness = finding.stale && finding.staleEvidence ? ` [STALE: ${finding.staleReason || 'anchor evidence changed'}]` - : ''; + : finding.stale + ? ' [possibly stale; verify before use]' + : ''; return `- ${finding.key}${freshness}: ${finding.claim.slice(0, 90)}`; }); return `# Project wiki (${total} findings, ${findings.length} listed) @@ -895,7 +905,7 @@ ${lines.join('\n')}`; * is the one that already happened here: an always-on block that grows until it * is wallpaper, and the model stops reading the thing it always sees. */ -export function standingRules(dir, graph, { budget = standingBudget() } = {}) { +export function standingRules(dir, graph, { budget = standingBudget(), episode = {} } = {}) { const rules = [...graph.nodes.values()].filter( (n) => n.kind === 'finding' && @@ -953,6 +963,7 @@ export function standingRules(dir, graph, { budget = standingBudget() } = {}) { // measurement shows no sign of it. const truncated = noticeFor(dropped); record(dir, { + ...episode, kind: 'standing', count: lines.length, dropped, diff --git a/plugin/hooks/lib/metrics.mjs b/plugin/hooks/lib/metrics.mjs index 3e4c4bb59..f6ab65780 100644 --- a/plugin/hooks/lib/metrics.mjs +++ b/plugin/hooks/lib/metrics.mjs @@ -761,8 +761,12 @@ export function report(dir) { // file 95 historical command injections into the file-read balance -- exactly // the dilution the split exists to prevent. const isCommand = (e) => e.surface === 'command' || e.trigger === 'command'; - const commandInjections = allInjections.filter(isCommand); - const injections = allInjections.filter((e) => !isCommand(e)); + // Session-start delivery has no file anchor or downstream read join either. + // Report its cost independently so it cannot dilute the file-touch estimate. + const isSessionStart = (e) => e.surface === 'session-start'; + const sessionStartInjections = allInjections.filter(isSessionStart); + const commandInjections = allInjections.filter((e) => !isSessionStart(e) && isCommand(e)); + const injections = allInjections.filter((e) => !isSessionStart(e) && !isCommand(e)); const treated = injections.filter((e) => !e.holdout); const withheld = injections.filter((e) => e.holdout); @@ -840,6 +844,10 @@ export function report(dir) { return { injections: treated.length, + sessionStartInjections: sessionStartInjections.length, + sessionStartInjectedTokens: sessionStartInjections.reduce( + (sum, event) => sum + (event.deliveredTokens ?? event.tokens ?? 0), 0 + ), commandInjections: commandInjections.length, commandHoldouts: commandInjections.filter((e) => e.holdout).length, holdouts: withheld.length, @@ -1080,7 +1088,9 @@ function matchesFilters(event, filters) { if (!filters[field]) continue; let actual = event[field]; if (event.kind === 'handoff-run') { - actual = field === 'taskId' ? event.scenarioId : (event.consumer?.[field] ?? event.producer?.[field]); + actual = field === 'taskId' + ? event.scenarioId + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); } @@ -1098,6 +1108,10 @@ function handoffArmMetrics(runs) { ); return { runs: runs.length, + delivery: proportionInterval( + runs.filter((run) => run.delivery?.delivered === true).length, + runs.length + ), correctness: proportionInterval(count('correct'), runs.length), firstPass: proportionInterval(count('firstPass'), runs.length), mistakeAttempted: proportionInterval(count('mistakeAttempted'), runs.length), @@ -1146,6 +1160,10 @@ const handoffKey = (run) => [ run.scenarioId || 'unknown', ].join('|'); +const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; +const handoffMinimumPairs = () => + Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + function handoffCohorts(runs) { const grouped = new Map(); for (const run of runs) { @@ -1153,7 +1171,7 @@ function handoffCohorts(runs) { if (!grouped.has(key)) grouped.set(key, []); grouped.get(key).push(run); } - const minimumPairs = Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10); + const minimumPairs = handoffMinimumPairs(); return [...grouped.entries()].map(([key, rows]) => { const arms = Object.fromEntries(HANDOFF_REPORT_ARMS.map((arm) => [ arm, handoffArmMetrics(rows.filter((row) => row.arm === arm)), @@ -1169,14 +1187,18 @@ function handoffCohorts(runs) { : null; const emptyRecurrence = arms.empty.mistakeExecuted.rate; const naturalRecurrence = arms.natural.mistakeExecuted.rate; - const relativeRecurrenceReduction = emptyRecurrence + const relativeRecurrenceReduction = emptyRecurrence !== null + && naturalRecurrence !== null + && emptyRecurrence !== 0 ? (emptyRecurrence - naturalRecurrence) / emptyRecurrence : null; const controlsSafe = ['irrelevant', 'stale'].every((arm) => - arms[arm].correctness.rate === null - || arms.empty.correctness.rate === null - || arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 + arms[arm].correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms[arm].correctness.rate >= arms.empty.correctness.rate - 0.1 ); + const irrelevantSuppressed = arms.irrelevant.delivery.rate !== null + && arms.irrelevant.delivery.rate === 0; const gates = { minimumPairs: naturalVsEmpty.pairs >= minimumPairs, capture: captureRate !== null && captureRate >= 0.8, @@ -1184,9 +1206,11 @@ function handoffCohorts(runs) { relativeRecurrenceReduction !== null && relativeRecurrenceReduction >= 0.5, recurrenceInterval: (naturalVsEmpty.executedMistakesPrevented.low ?? -Infinity) > 0, correctness: - (arms.natural.correctness.rate ?? -Infinity) >= (arms.empty.correctness.rate ?? 0) - 0.1, + arms.natural.correctness.rate !== null + && arms.empty.correctness.rate !== null + && arms.natural.correctness.rate >= arms.empty.correctness.rate - 0.1, preActionDelivery: preActionDeliveryRate !== null && preActionDeliveryRate >= 0.8, - negativeControls: controlsSafe, + negativeControls: controlsSafe && irrelevantSuppressed, }; const claimReady = Object.values(gates).every(Boolean); return { @@ -1209,8 +1233,6 @@ function handoffCohorts(runs) { }); } -const HANDOFF_REPORT_ARMS = ['empty', 'natural', 'oracle', 'irrelevant', 'stale']; - function concurrencySummary(runs) { const natural = runs.filter((run) => run.arm === 'natural'); const writers = natural.reduce((sum, run) => sum + (run.writerCount || 0), 0); @@ -1345,8 +1367,7 @@ export function evidenceReport(dir, { filters = {}, episodeLimit = 100 } = {}) { causalRule: 'matched pairs estimate optimizer vs baseline, retrieval vs optimizer, full vs retrieval, and full vs baseline', minimumPairs, - handoffMinimumPairs: - Math.max(2, Number(process.env.TOKEN_OPTIMIZER_HANDOFF_MIN_PAIRS) || 10), + handoffMinimumPairs: handoffMinimumPairs(), deterministicChecksAreCausalProof: false, }, }; diff --git a/scripts/run-concurrent-handoff-eval.mjs b/scripts/run-concurrent-handoff-eval.mjs index 59efdcf36..2c8cfae51 100644 --- a/scripts/run-concurrent-handoff-eval.mjs +++ b/scripts/run-concurrent-handoff-eval.mjs @@ -11,7 +11,7 @@ import { dirname, isAbsolute, join, resolve } from 'node:path'; import { tmpdir } from 'node:os'; import { fileURLToPath } from 'node:url'; import { - extractToolTrace, findNaturalCapture, gradeBehavior, readJsonl, runClient, + extractToolTrace, findNaturalCapture, gradeBehavior, graphDigest, readJsonl, runClient, } from './run-handoff-eval.mjs'; import { parseRunIdentity, parseUsage } from './run-evidence-eval.mjs'; import { audit as auditGraph } from '../hooks-core/curate.mjs'; @@ -41,11 +41,16 @@ function options(argv) { graphDir: wikiDir(ROOT), keepWorkspaces: false, }; + const valueOptions = new Set([ + 'suite', 'repetitions', 'output', 'graphDir', 'runner', 'producer', + 'consumer', 'producerModel', 'consumerModel', + ]); for (let index = 0; index < argv.length; index++) { const arg = argv[index]; if (arg === '--keep-workspaces') out.keepWorkspaces = true; else if (arg.startsWith('--')) { const key = arg.slice(2).replace(/-([a-z])/g, (_, letter) => letter.toUpperCase()); + if (!valueOptions.has(key)) throw new Error(`unknown option: ${arg}`); const value = argv[++index]; if (value === undefined) throw new Error(`${arg} requires a value`); out[key] = value; @@ -138,15 +143,21 @@ async function main() { for (let repetition = 0; repetition < cli.repetitions; repetition++) { const pairId = `concurrent-${repetition + 1}`; const root = mkdtempSync(join(tmpdir(), `token-optimizer-${pairId}-`)); - const workspace = join(root, 'workspace'); - cpSync(fromRoot(scenarios[0].fixture), workspace, { recursive: true }); + try { const graphDir = join(root, 'shared-graph'); - const auditPath = join(root, 'producer-audit.jsonl'); mkdirSync(graphDir, { recursive: true }); + const producerWorkspaces = scenarios.map((scenario, index) => { + const workspace = join(root, `writer-${index}-${scenario.id}`); + cpSync(fromRoot(scenario.fixture), workspace, { recursive: true }); + return workspace; + }); + const producerAuditPaths = scenarios.map( + (scenario, index) => join(root, `writer-${index}-${scenario.id}-audit.jsonl`) + ); // Promise.all is deliberate: these are separate CLI processes appending to // the same graph and metrics logs, not sequential calls labeled concurrent. - const producers = await Promise.all(scenarios.map((scenario) => runClient({ + const producers = await Promise.all(scenarios.map((scenario, index) => runClient({ profile: producerProfile, client: cli.producer, model: cli.producerModel || producerProfile.model, @@ -154,11 +165,17 @@ async function main() { phase: `writer-${scenario.id}`, scenario, pairId, - workspace, + workspace: producerWorkspaces[index], graphDir, - auditPath, + auditPath: producerAuditPaths[index], }))); - const producerAudit = readJsonl(auditPath); + const producerAudits = producers.map((producer, index) => + readJsonl(producerAuditPaths[index]).filter((event) => + event.writerId === producer.episodeId + && event.scenarioId === scenarios[index].id + && event.phase === `writer-${scenarios[index].id}` + ) + ); const graph = loadGraph(graphDir); const captures = capturesByWriter(graph, scenarios, producers); const findingIds = [...new Set(captures.flat().map((finding) => finding.key))]; @@ -176,7 +193,13 @@ async function main() { clientVersion: identity.clientVersion || producerProfile.version || null, model: cli.producerModel || producerProfile.model || null, modelVersion: identity.modelVersion || producerProfile.modelVersion || null, - ...gradeBehavior(scenario, 'producer', workspace, producerAudit, producer.trace), + ...gradeBehavior( + scenario, + 'producer', + producerWorkspaces[index], + producerAudits[index], + producer.trace + ), captureSuccess: captures[index].length > 0, capturedFindingIds: captures[index].map((finding) => finding.key), acceptedFindingIds: producer.acceptedFindingIds, @@ -191,7 +214,7 @@ async function main() { }); const integrity = graphIntegrity(graphDir, acceptedFindingIds); const frozen = join(root, 'frozen'); - cpSync(workspace, frozen, { recursive: true }); + cpSync(fromRoot(scenarios[0].fixture), frozen, { recursive: true }); const arms = repetition % 2 ? ['natural', 'empty'] : ['empty', 'natural']; for (const [order, arm] of arms.entries()) { @@ -218,9 +241,7 @@ async function main() { const behaviors = scenarios.map((scenario) => gradeBehavior(scenario, 'consumer', consumerWorkspace, consumerAudit, consumer.trace) ); - const deliveryEvents = consumer.evidence.filter((event) => - event.kind === 'inject' && event.surface === 'session-start' - ); + const deliveryEvents = consumer.evidence.filter((event) => event.kind === 'inject'); const deliveredIds = new Set(deliveryEvents.flatMap((event) => event.findingIds || [])); const identity = parseRunIdentity(consumer.run.stdout); const usage = parseUsage( @@ -267,18 +288,14 @@ async function main() { delivered: arm === 'natural' ? findingIds.filter((id) => deliveredIds.has(id)).length : 0, - allBeforeFirstAction: arm === 'natural' + allExpectedFindingsDelivered: arm === 'natural' && findingIds.length ? findingIds.every((id) => deliveredIds.has(id)) : null, deliveredTokens: deliveryEvents.reduce( (sum, event) => sum + (event.deliveredTokens || 0), 0 ), }, - graphHash: hash( - existsSync(join(graphDir, 'graph.jsonl')) - ? readFileSync(join(graphDir, 'graph.jsonl'), 'utf8') - : 'empty' - ), + graphHash: graphDigest(graphDir), producerPromptHashes: scenarios.map((scenario) => hash(scenario.producerPrompt)), consumerPromptHash: hash(combinedPrompt()), repoCommit: process.env.GITHUB_SHA || PROVENANCE.commit, @@ -294,7 +311,9 @@ async function main() { ); } - if (!cli.keepWorkspaces) rmSync(root, { recursive: true, force: true }); + } finally { + if (!cli.keepWorkspaces) rmSync(root, { recursive: true, force: true }); + } } process.stdout.write(`Concurrency evidence artifact: ${output}\n`); } diff --git a/scripts/run-evidence-eval.mjs b/scripts/run-evidence-eval.mjs index bcb6830c7..2b8b620e9 100644 --- a/scripts/run-evidence-eval.mjs +++ b/scripts/run-evidence-eval.mjs @@ -212,26 +212,33 @@ export function parseUsage(text, profile = {}) { export function parseRunIdentity(text) { let identity = { modelVersion: null, clientVersion: null, sessionId: null }; + const firstPresent = (...values) => values.find( + (value) => value !== undefined && value !== null && value !== '' + ) ?? null; for (const line of String(text).split('\n')) { try { const document = JSON.parse(line); if (document?.type === 'system' && document?.subtype === 'init') { identity = { - modelVersion: document.model || null, - clientVersion: document.claude_code_version || null, - sessionId: document.session_id || null, + modelVersion: firstPresent(identity.modelVersion, document.model), + clientVersion: firstPresent(identity.clientVersion, document.claude_code_version), + sessionId: firstPresent(identity.sessionId, document.session_id), }; } // Codex JSONL identifies the ephemeral thread separately from turn // events. Keep scanning because future versions may add explicit model // or client metadata later in the stream. if (document?.type === 'thread.started') { - identity.sessionId = document.thread_id || document.threadId || identity.sessionId; + identity.sessionId = firstPresent( + identity.sessionId, document.thread_id, document.threadId + ); } - identity.modelVersion = document.model_version || document.modelVersion - || document.model || identity.modelVersion; - identity.clientVersion = document.client_version || document.clientVersion - || identity.clientVersion; + identity.modelVersion = firstPresent( + identity.modelVersion, document?.model_version, document?.modelVersion, document?.model + ); + identity.clientVersion = firstPresent( + identity.clientVersion, document?.client_version, document?.clientVersion + ); } catch { /* non-JSON output */ } } return identity; diff --git a/scripts/run-handoff-eval.mjs b/scripts/run-handoff-eval.mjs index 401f14754..634377a24 100644 --- a/scripts/run-handoff-eval.mjs +++ b/scripts/run-handoff-eval.mjs @@ -51,6 +51,10 @@ function parseArgs(argv) { keepWorkspaces: false, includeTranscript: false, }; + const valueOptions = new Set([ + 'suite', 'repetitions', 'output', 'graphDir', 'scenarios', 'arms', + 'runner', 'producer', 'consumer', 'producerModel', 'consumerModel', + ]); for (let index = 0; index < argv.length; index++) { const arg = argv[index]; if (arg === '--dry-run') options.dryRun = true; @@ -58,6 +62,7 @@ function parseArgs(argv) { else if (arg === '--include-transcript') options.includeTranscript = true; else if (arg.startsWith('--')) { const key = arg.slice(2).replace(/-([a-z])/g, (_, letter) => letter.toUpperCase()); + if (!valueOptions.has(key)) throw new Error(`unknown option: ${arg}`); const value = argv[++index]; if (value === undefined) throw new Error(`${arg} requires a value`); options[key] = value; @@ -100,23 +105,31 @@ function execute(command, args, { cwd, env, timeoutMs }) { const stdout = []; const stderr = []; let timedOut = false; + let killTimer = null; + let settled = false; + const settle = (result) => { + if (settled) return; + settled = true; + clearTimeout(timer); + if (killTimer) clearTimeout(killTimer); + finish(result); + }; const timer = setTimeout(() => { timedOut = true; child.kill(); + killTimer = setTimeout(() => child.kill('SIGKILL'), 5_000); }, timeoutMs); child.stdout.on('data', (chunk) => stdout.push(chunk)); child.stderr.on('data', (chunk) => stderr.push(chunk)); child.on('error', (error) => { - clearTimeout(timer); - finish({ + settle({ exitCode: null, signal: null, timedOut, latencyMs: Date.now() - started, stdout: Buffer.concat(stdout).toString('utf8'), stderr: `${Buffer.concat(stderr).toString('utf8')}\n${error.message}`.trim(), }); }); child.on('close', (exitCode, signal) => { - clearTimeout(timer); - finish({ + settle({ exitCode, signal, timedOut, latencyMs: Date.now() - started, stdout: Buffer.concat(stdout).toString('utf8'), stderr: Buffer.concat(stderr).toString('utf8'), @@ -272,6 +285,10 @@ function readFixtureJson(workspace, path) { try { return JSON.parse(readFileSync(join(workspace, path), 'utf8')); } catch { return null; } } +function readFixtureText(workspace, path) { + try { return readFileSync(join(workspace, path), 'utf8').trim(); } catch { return null; } +} + /** Hidden behavioral grader; it never scores claims made in the final answer. */ export function gradeBehavior(scenario, phase, workspace, audit, trace) { const target = phase === 'producer' @@ -299,8 +316,8 @@ export function gradeBehavior(scenario, phase, workspace, audit, trace) { const synchronized = audit.some( (event) => event.kind === 'sync-check' && event.synchronized === true ); - const source = readFileSync(join(workspace, 'source', `${target}-policy.txt`), 'utf8').trim(); - const client = readFileSync(join(workspace, 'clients', target, 'policy.txt'), 'utf8').trim(); + const source = readFixtureText(workspace, `source/${target}-policy.txt`); + const client = readFixtureText(workspace, `clients/${target}/policy.txt`); proof = synchronized; correct = source === 'modern' && client === 'modern' && synchronized; } else if (scenario.grader.kind === 'validation') { @@ -374,7 +391,7 @@ function seedArm(arm, scenario, workspace, graphDir) { return keys; } -function graphDigest(dir) { +export function graphDigest(dir) { if (!existsSync(dir)) return hash('empty'); const files = []; const visit = (current) => { @@ -424,6 +441,9 @@ export async function runClient({ TOKEN_OPTIMIZER_SHARED_DIR: graphDir, TOKEN_OPTIMIZER_STATE_DIR: stateDir, TOKEN_OPTIMIZER_EVAL_AUDIT: auditPath, + TOKEN_OPTIMIZER_EVAL_WRITER_ID: episodeId, + TOKEN_OPTIMIZER_EVAL_SCENARIO_ID: scenario.id, + TOKEN_OPTIMIZER_EVAL_PHASE: phase, }; const run = await execute(profile.command, args, { cwd: workspace, @@ -453,7 +473,7 @@ function deliveryGrade(evidence, findingIds, audit, behavior) { deliveredTokens: relevant.reduce((sum, event) => sum + (event.deliveredTokens || 0), 0), beforeFirstExecutedMistake: first ? (firstMistakeAt === null || Number(first.at) <= Number(firstMistakeAt)) - : false, + : null, }; } @@ -518,6 +538,7 @@ async function main() { for (const item of schedule) { const pairRoot = mkdtempSync(join(tmpdir(), `token-optimizer-handoff-${item.pairId}-`)); const producerWorkspace = materialize(item.scenario.fixture, `producer-${item.pairId}-`); + try { const producerGraph = join(pairRoot, 'producer-graph'); const producerAuditPath = join(pairRoot, 'producer-audit.jsonl'); mkdirSync(producerGraph, { recursive: true }); @@ -637,7 +658,9 @@ async function main() { workingTreeDirty: PROVENANCE.dirty, producerPromptHash: hash(item.scenario.producerPrompt), consumerPromptHash: hash(item.scenario.consumerPrompt), - fixtureHash: hash(JSON.stringify(item.scenario.grader) + item.scenario.fixture), + fixtureHash: hash( + JSON.stringify(item.scenario.grader) + graphDigest(fromRoot(item.scenario.fixture)) + ), stdoutHash: hash(consumer.run.stdout), stderrHash: hash(consumer.run.stderr), ...(options.includeTranscript @@ -657,8 +680,10 @@ async function main() { ); } - rmSync(producerWorkspace, { recursive: true, force: true }); - if (!options.keepWorkspaces) rmSync(pairRoot, { recursive: true, force: true }); + } finally { + rmSync(producerWorkspace, { recursive: true, force: true }); + if (!options.keepWorkspaces) rmSync(pairRoot, { recursive: true, force: true }); + } } process.stdout.write(`Handoff evidence artifact: ${output}\n`); } diff --git a/scripts/verify-wiki-ui.mjs b/scripts/verify-wiki-ui.mjs index d04bb5afe..fb6621679 100644 --- a/scripts/verify-wiki-ui.mjs +++ b/scripts/verify-wiki-ui.mjs @@ -125,8 +125,8 @@ async function seed() { integrity: { zeroLoss: true, parseable: true, orphanedFindings: 0 }, delivery: { expected: arm === 'natural' ? 3 : 0, delivered: arm === 'natural' ? 3 : 0 }, consumer: { - client: 'claude-code', model: 'claude-sonnet-5', correct: true, - firstPass: arm === 'natural', mistakeExecuted: arm === 'empty', + client: 'claude-code', model: 'claude-sonnet-5', correct: arm === 'empty', + firstPass: false, mistakeAttempted: true, mistakeExecuted: true, }, }); } @@ -343,7 +343,8 @@ async function main() { && /gates passed/i.test(transferText || '')); const concurrencyText = await page.textContent('#evidence-concurrency'); check('evidence renders concurrent writer integrity and later delivery', - /100%/.test(concurrencyText || '') && /3/.test(concurrencyText || '')); + /100%/.test(concurrencyText || '') && /0%/.test(concurrencyText || '') + && /3/.test(concurrencyText || '')); const capabilityCount = await page.locator('#evidence-capabilities tbody tr').count(); check('evidence lists all supported client capabilities', capabilityCount === 16, String(capabilityCount)); const capabilityText = await page.textContent('#evidence-capabilities'); diff --git a/tests/hooks/causal-evidence.test.mjs b/tests/hooks/causal-evidence.test.mjs index cb81f3a7d..537945707 100644 --- a/tests/hooks/causal-evidence.test.mjs +++ b/tests/hooks/causal-evidence.test.mjs @@ -194,7 +194,7 @@ describe('paired evidence report', () => { latencyMs: prevented ? 8000 : 10000, }, delivery: { - delivered: arm !== 'empty', + delivered: ['natural', 'oracle', 'stale'].includes(arm), beforeFirstExecutedMistake: arm === 'natural', }, }); @@ -210,9 +210,47 @@ describe('paired evidence report', () => { expect(cohort.arms.natural.mistakeExecuted.rate).toBe(0); expect(cohort.effects.naturalVsEmpty.executedMistakesPrevented.low).toBe(1); expect(cohort.effects.naturalVsOracle.executedMistakesPrevented.mean).toBe(0); + expect(cohort.arms.irrelevant.delivery.rate).toBe(0); expect(cohort.evidenceStatus).toBe('pre-registered transfer gates passed'); }); + test('fails transfer gates when control evidence is absent or irrelevant content is delivered', () => { + for (let pair = 1; pair <= 10; pair++) { + for (const arm of ['empty', 'natural', 'irrelevant']) { + record(dir, { + kind: 'handoff-run', pairId: `negative-${pair}`, + scenarioId: 'verification-entry-point', arm, + producer: { client: 'codex', model: 'gpt-5.6-sol', captureSuccess: true }, + consumer: { + client: 'claude-code', model: 'claude-sonnet-5', correct: true, + mistakeExecuted: arm === 'empty', + }, + delivery: { + delivered: arm !== 'empty', + beforeFirstExecutedMistake: arm === 'natural', + }, + }); + } + } + + const cohort = evidenceReport(dir).transferCohorts[0]; + expect(cohort.gates.negativeControls).toBe(false); + expect(cohort.evidenceStatus).toMatch(/insufficient or failed/); + }); + + test('filters handoff arms from the top-level arm field', () => { + for (const arm of ['empty', 'natural']) { + record(dir, { + kind: 'handoff-run', pairId: 'filter-1', scenarioId: 'filter-task', arm, + producer: { client: 'codex' }, consumer: { client: 'claude-code' }, + }); + } + const filtered = evidenceReport(dir, { filters: { arm: 'natural' } }); + expect(filtered.summary.handoffRuns).toBe(1); + expect(filtered.transferCohorts[0].arms.natural.runs).toBe(1); + expect(filtered.transferCohorts[0].arms.empty.runs).toBe(0); + }); + test('reports concurrent graph integrity separately from consumer behavior', () => { for (const arm of ['empty', 'natural']) { record(dir, { diff --git a/tests/hooks/evidence-harness.test.mjs b/tests/hooks/evidence-harness.test.mjs index d55c5c60a..195848633 100644 --- a/tests/hooks/evidence-harness.test.mjs +++ b/tests/hooks/evidence-harness.test.mjs @@ -36,4 +36,21 @@ describe('live evidence harness', () => { modelVersion: 'claude-sonnet-5', clientVersion: '2.1.225', sessionId: 's1', }); }); + + test('does not let later sparse events overwrite the authoritative run identity', () => { + const stream = [ + JSON.stringify({ + type: 'system', subtype: 'init', model: 'model-first', + claude_code_version: 'client-first', session_id: 'session-first', + }), + JSON.stringify({ + type: 'thread.started', thread_id: 'session-later', + model_version: 'model-later', client_version: 'client-later', + }), + ].join('\n'); + + expect(parseRunIdentity(stream)).toEqual({ + modelVersion: 'model-first', clientVersion: 'client-first', sessionId: 'session-first', + }); + }); }); diff --git a/tests/hooks/handoff-evidence.test.mjs b/tests/hooks/handoff-evidence.test.mjs index 19c7d9937..bf72082fc 100644 --- a/tests/hooks/handoff-evidence.test.mjs +++ b/tests/hooks/handoff-evidence.test.mjs @@ -1,5 +1,5 @@ import { afterEach, beforeEach, describe, expect, test } from '@jest/globals'; -import { cpSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'; +import { cpSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; import { join, resolve } from 'node:path'; import { tmpdir } from 'node:os'; import { @@ -12,6 +12,7 @@ import { targetFindingMatches, } from '../../scripts/run-handoff-eval.mjs'; import { capturesByWriter } from '../../scripts/run-concurrent-handoff-eval.mjs'; +import { audit as writeFixtureAudit } from '../../evals/fixtures/mistake-transfer/scripts/audit.mjs'; const ROOT = resolve(process.cwd()); const suite = JSON.parse( @@ -126,6 +127,16 @@ describe('hidden behavioral graders', () => { expect(direct).toMatchObject({ correct: true, firstPass: false, mistakeExecuted: true }); }); + test('generated-file grading fails closed when a generated or source file is missing', () => { + const scenario = suite.scenarios.find((item) => item.grader.kind === 'generated'); + rmSync(join(workspace, 'source', 'beta-policy.txt'), { force: true }); + + expect(() => gradeBehavior(scenario, 'consumer', workspace, [], [])).not.toThrow(); + expect(gradeBehavior(scenario, 'consumer', workspace, [], [])).toMatchObject({ + correct: false, proof: false, + }); + }); + test('validation requires the scoped sentinel and rejects a zero-target PASS as first-pass', () => { const scenario = suite.scenarios.find((item) => item.grader.kind === 'validation'); writeFileSync(join(workspace, 'targets', 'beta.json'), '{"ready":true}\n'); @@ -141,6 +152,38 @@ describe('hidden behavioral graders', () => { }); }); +describe('fixture audit provenance', () => { + test('uses runner-owned provenance and an authoritative timestamp', () => { + const auditPath = join(workspace, 'audit.jsonl'); + const keys = [ + 'TOKEN_OPTIMIZER_EVAL_AUDIT', 'TOKEN_OPTIMIZER_EVAL_WRITER_ID', + 'TOKEN_OPTIMIZER_EVAL_SCENARIO_ID', 'TOKEN_OPTIMIZER_EVAL_PHASE', + ]; + const before = Object.fromEntries(keys.map((key) => [key, process.env[key]])); + try { + process.env.TOKEN_OPTIMIZER_EVAL_AUDIT = auditPath; + process.env.TOKEN_OPTIMIZER_EVAL_WRITER_ID = 'writer-real'; + process.env.TOKEN_OPTIMIZER_EVAL_SCENARIO_ID = 'scenario-real'; + process.env.TOKEN_OPTIMIZER_EVAL_PHASE = 'producer'; + writeFixtureAudit({ + kind: 'sentinel', at: 1, writerId: 'writer-spoofed', + scenarioId: 'scenario-spoofed', phase: 'consumer', + }); + const row = JSON.parse(readFileSync(auditPath, 'utf8').trim()); + expect(row).toMatchObject({ + kind: 'sentinel', writerId: 'writer-real', + scenarioId: 'scenario-real', phase: 'producer', + }); + expect(row.at).toBeGreaterThan(1); + } finally { + for (const key of keys) { + if (before[key] === undefined) delete process.env[key]; + else process.env[key] = before[key]; + } + } + }); +}); + describe('CLI trace normalization', () => { test('reads Claude tool_use and Codex command_execution events without final prose', () => { const stream = [ diff --git a/tests/hooks/holdout-measurement.test.mjs b/tests/hooks/holdout-measurement.test.mjs index a683f345e..bf42fb724 100644 --- a/tests/hooks/holdout-measurement.test.mjs +++ b/tests/hooks/holdout-measurement.test.mjs @@ -183,4 +183,20 @@ describe('the command path takes part in the holdout', () => { expect(r.commandInjections).toBe(2); expect(r.commandHoldouts).toBe(1); }); + + it('keeps session-start delivery out of the file-read balance', () => { + record(dir, { + kind: 'inject', surface: 'session-start', anchor: 'session-index', + holdout: false, tokens: 80, deliveredTokens: 80, + }); + record(dir, { + kind: 'inject', surface: 'file', anchor: '/a.ts', holdout: false, tokens: 20, + }); + + const r = report(dir); + expect(r.injections).toBe(1); + expect(r.injectedTokens).toBe(20); + expect(r.sessionStartInjections).toBe(1); + expect(r.sessionStartInjectedTokens).toBe(80); + }); }); diff --git a/tests/hooks/injection.test.mjs b/tests/hooks/injection.test.mjs index 117831a19..a6db68350 100644 --- a/tests/hooks/injection.test.mjs +++ b/tests/hooks/injection.test.mjs @@ -142,6 +142,7 @@ describe('P4 injection', () => { const index = sessionIndex(dir, load(dir), { episode: { episodeId: 'consumer-1', client: 'codex', pairId: 'pair-1' }, + relevantFindingIds: ['the retry bu'], }); expect(index).toContain('retry budget'); expect(index).toContain('wiki_query'); @@ -165,7 +166,9 @@ describe('P4 injection', () => { seedFinding(path, 'the old mode is always required'); writeFileSync(path, 'export const mode = "new";'); - const index = sessionIndex(dir, load(dir, { snapshots: true })); + const graph = load(dir, { snapshots: true }); + const key = [...graph.nodes.values()].find((node) => node.kind === 'finding').key; + const index = sessionIndex(dir, graph, { relevantFindingIds: [key] }); expect(index).toContain('STALE: file changed'); expect(index).toContain('old mode'); }); diff --git a/tests/hooks/standing-rules.test.mjs b/tests/hooks/standing-rules.test.mjs index 0df594d4a..595496d4e 100644 --- a/tests/hooks/standing-rules.test.mjs +++ b/tests/hooks/standing-rules.test.mjs @@ -11,7 +11,7 @@ * is narrow and the budget is fixed rather than earned. */ import { describe, it, expect, beforeEach, afterEach } from '@jest/globals'; -import { standingRules } from '../../hooks-core/inject.mjs'; +import { sessionIndex, standingRules } from '../../hooks-core/inject.mjs'; import { load, putNode, wikiDir } from '../../hooks-core/wiki.mjs'; import { spawnSync } from 'child_process'; import { mkdtempSync, rmSync, readFileSync, mkdirSync } from 'fs'; @@ -134,7 +134,7 @@ describe('the budget', () => { }); describe('through the real SessionStart hook', () => { - it('delivers standing rules and the bounded project index before the first tool call', () => { + it('delivers universal rules but withholds situational findings until relevance exists', () => { // The unit tests prove the selection. This proves the hook actually emits // it -- the distinction that mattered for forTouch, which was correct and // called by nothing for its entire life. @@ -173,12 +173,13 @@ describe('through the real SessionStart hook', () => { expect(ctx).toContain('# Standing rules'); expect(ctx).toContain('Report the number you measured'); expect(ctx).toContain('isolated worktree'); - // A situational finding is catalogued, not promoted into the full standing - // rules block. This makes it available before the first action while the - // tighter block remains limited to genuinely universal rules. - expect(ctx).toContain('# Project wiki'); - expect(ctx).toContain('npx jest'); - expect(ctx.indexOf('# Project wiki')).toBeGreaterThan(ctx.indexOf('# Standing rules')); + expect(ctx).not.toContain('# Project wiki'); + expect(ctx).not.toContain('npx jest'); + + // A prompt/tool relevance pass can explicitly select the situational item. + const relevant = sessionIndex(graphDir, load(graphDir), { relevantFindingIds: ['c1'] }); + expect(relevant).toContain('# Project wiki'); + expect(relevant).toContain('npx jest'); try { rmSync(project, { recursive: true, force: true }); } catch { /* windows */ } }); From 521d257551db4a6a126c078d6c85f73b7d83bd50 Mon Sep 17 00:00:00 2001 From: Franklin Moormann Date: Sun, 9 Aug 2026 14:08:53 -0400 Subject: [PATCH 8/9] test: keep experiment arm authoritative --- hooks-core/metrics.mjs | 10 ++++++++-- .../cline/hooks/token-optimizer/lib/metrics.mjs | 10 ++++++++-- integrations/codex/hooks/lib/metrics.mjs | 10 ++++++++-- integrations/codex/plugin/hooks/lib/metrics.mjs | 10 ++++++++-- integrations/copilot/.github/hooks/lib/metrics.mjs | 10 ++++++++-- integrations/cursor/hooks/lib/metrics.mjs | 10 ++++++++-- integrations/gemini/hooks/lib/metrics.mjs | 10 ++++++++-- integrations/kilo/hooks/lib/metrics.mjs | 10 ++++++++-- integrations/opencode/hooks/lib/metrics.mjs | 10 ++++++++-- integrations/qwen/hooks/lib/metrics.mjs | 10 ++++++++-- integrations/windsurf/hooks/lib/metrics.mjs | 10 ++++++++-- plugin/hooks/lib/metrics.mjs | 10 ++++++++-- tests/hooks/causal-evidence.test.mjs | 3 ++- 13 files changed, 98 insertions(+), 25 deletions(-) diff --git a/hooks-core/metrics.mjs b/hooks-core/metrics.mjs index 7e29b2c4c..8d75f342c 100644 --- a/hooks-core/metrics.mjs +++ b/hooks-core/metrics.mjs @@ -1088,9 +1088,15 @@ function matchesFilters(event, filters) { if (event.kind === 'handoff-run') { actual = field === 'taskId' ? event.scenarioId - : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { - actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + actual = field === 'taskId' + ? 'concurrent-combined' + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event[field]); } if (String(actual || '') !== String(filters[field])) return false; } diff --git a/integrations/cline/hooks/token-optimizer/lib/metrics.mjs b/integrations/cline/hooks/token-optimizer/lib/metrics.mjs index f6ab65780..913e2af66 100644 --- a/integrations/cline/hooks/token-optimizer/lib/metrics.mjs +++ b/integrations/cline/hooks/token-optimizer/lib/metrics.mjs @@ -1090,9 +1090,15 @@ function matchesFilters(event, filters) { if (event.kind === 'handoff-run') { actual = field === 'taskId' ? event.scenarioId - : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { - actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + actual = field === 'taskId' + ? 'concurrent-combined' + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event[field]); } if (String(actual || '') !== String(filters[field])) return false; } diff --git a/integrations/codex/hooks/lib/metrics.mjs b/integrations/codex/hooks/lib/metrics.mjs index f6ab65780..913e2af66 100644 --- a/integrations/codex/hooks/lib/metrics.mjs +++ b/integrations/codex/hooks/lib/metrics.mjs @@ -1090,9 +1090,15 @@ function matchesFilters(event, filters) { if (event.kind === 'handoff-run') { actual = field === 'taskId' ? event.scenarioId - : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { - actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + actual = field === 'taskId' + ? 'concurrent-combined' + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event[field]); } if (String(actual || '') !== String(filters[field])) return false; } diff --git a/integrations/codex/plugin/hooks/lib/metrics.mjs b/integrations/codex/plugin/hooks/lib/metrics.mjs index f6ab65780..913e2af66 100644 --- a/integrations/codex/plugin/hooks/lib/metrics.mjs +++ b/integrations/codex/plugin/hooks/lib/metrics.mjs @@ -1090,9 +1090,15 @@ function matchesFilters(event, filters) { if (event.kind === 'handoff-run') { actual = field === 'taskId' ? event.scenarioId - : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { - actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + actual = field === 'taskId' + ? 'concurrent-combined' + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event[field]); } if (String(actual || '') !== String(filters[field])) return false; } diff --git a/integrations/copilot/.github/hooks/lib/metrics.mjs b/integrations/copilot/.github/hooks/lib/metrics.mjs index f6ab65780..913e2af66 100644 --- a/integrations/copilot/.github/hooks/lib/metrics.mjs +++ b/integrations/copilot/.github/hooks/lib/metrics.mjs @@ -1090,9 +1090,15 @@ function matchesFilters(event, filters) { if (event.kind === 'handoff-run') { actual = field === 'taskId' ? event.scenarioId - : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { - actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + actual = field === 'taskId' + ? 'concurrent-combined' + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event[field]); } if (String(actual || '') !== String(filters[field])) return false; } diff --git a/integrations/cursor/hooks/lib/metrics.mjs b/integrations/cursor/hooks/lib/metrics.mjs index f6ab65780..913e2af66 100644 --- a/integrations/cursor/hooks/lib/metrics.mjs +++ b/integrations/cursor/hooks/lib/metrics.mjs @@ -1090,9 +1090,15 @@ function matchesFilters(event, filters) { if (event.kind === 'handoff-run') { actual = field === 'taskId' ? event.scenarioId - : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { - actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + actual = field === 'taskId' + ? 'concurrent-combined' + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event[field]); } if (String(actual || '') !== String(filters[field])) return false; } diff --git a/integrations/gemini/hooks/lib/metrics.mjs b/integrations/gemini/hooks/lib/metrics.mjs index f6ab65780..913e2af66 100644 --- a/integrations/gemini/hooks/lib/metrics.mjs +++ b/integrations/gemini/hooks/lib/metrics.mjs @@ -1090,9 +1090,15 @@ function matchesFilters(event, filters) { if (event.kind === 'handoff-run') { actual = field === 'taskId' ? event.scenarioId - : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { - actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + actual = field === 'taskId' + ? 'concurrent-combined' + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event[field]); } if (String(actual || '') !== String(filters[field])) return false; } diff --git a/integrations/kilo/hooks/lib/metrics.mjs b/integrations/kilo/hooks/lib/metrics.mjs index f6ab65780..913e2af66 100644 --- a/integrations/kilo/hooks/lib/metrics.mjs +++ b/integrations/kilo/hooks/lib/metrics.mjs @@ -1090,9 +1090,15 @@ function matchesFilters(event, filters) { if (event.kind === 'handoff-run') { actual = field === 'taskId' ? event.scenarioId - : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { - actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + actual = field === 'taskId' + ? 'concurrent-combined' + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event[field]); } if (String(actual || '') !== String(filters[field])) return false; } diff --git a/integrations/opencode/hooks/lib/metrics.mjs b/integrations/opencode/hooks/lib/metrics.mjs index f6ab65780..913e2af66 100644 --- a/integrations/opencode/hooks/lib/metrics.mjs +++ b/integrations/opencode/hooks/lib/metrics.mjs @@ -1090,9 +1090,15 @@ function matchesFilters(event, filters) { if (event.kind === 'handoff-run') { actual = field === 'taskId' ? event.scenarioId - : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { - actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + actual = field === 'taskId' + ? 'concurrent-combined' + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event[field]); } if (String(actual || '') !== String(filters[field])) return false; } diff --git a/integrations/qwen/hooks/lib/metrics.mjs b/integrations/qwen/hooks/lib/metrics.mjs index f6ab65780..913e2af66 100644 --- a/integrations/qwen/hooks/lib/metrics.mjs +++ b/integrations/qwen/hooks/lib/metrics.mjs @@ -1090,9 +1090,15 @@ function matchesFilters(event, filters) { if (event.kind === 'handoff-run') { actual = field === 'taskId' ? event.scenarioId - : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { - actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + actual = field === 'taskId' + ? 'concurrent-combined' + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event[field]); } if (String(actual || '') !== String(filters[field])) return false; } diff --git a/integrations/windsurf/hooks/lib/metrics.mjs b/integrations/windsurf/hooks/lib/metrics.mjs index f6ab65780..913e2af66 100644 --- a/integrations/windsurf/hooks/lib/metrics.mjs +++ b/integrations/windsurf/hooks/lib/metrics.mjs @@ -1090,9 +1090,15 @@ function matchesFilters(event, filters) { if (event.kind === 'handoff-run') { actual = field === 'taskId' ? event.scenarioId - : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { - actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + actual = field === 'taskId' + ? 'concurrent-combined' + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event[field]); } if (String(actual || '') !== String(filters[field])) return false; } diff --git a/plugin/hooks/lib/metrics.mjs b/plugin/hooks/lib/metrics.mjs index f6ab65780..913e2af66 100644 --- a/plugin/hooks/lib/metrics.mjs +++ b/plugin/hooks/lib/metrics.mjs @@ -1090,9 +1090,15 @@ function matchesFilters(event, filters) { if (event.kind === 'handoff-run') { actual = field === 'taskId' ? event.scenarioId - : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event.producer?.[field] ?? event[field]); } else if (event.kind === 'concurrency-run') { - actual = field === 'taskId' ? 'concurrent-combined' : (event.consumer?.[field] ?? event[field]); + actual = field === 'taskId' + ? 'concurrent-combined' + : field === 'arm' + ? event.arm + : (event.consumer?.[field] ?? event[field]); } if (String(actual || '') !== String(filters[field])) return false; } diff --git a/tests/hooks/causal-evidence.test.mjs b/tests/hooks/causal-evidence.test.mjs index 537945707..9c8196729 100644 --- a/tests/hooks/causal-evidence.test.mjs +++ b/tests/hooks/causal-evidence.test.mjs @@ -242,7 +242,8 @@ describe('paired evidence report', () => { for (const arm of ['empty', 'natural']) { record(dir, { kind: 'handoff-run', pairId: 'filter-1', scenarioId: 'filter-task', arm, - producer: { client: 'codex' }, consumer: { client: 'claude-code' }, + producer: { client: 'codex', arm: 'nested-producer-value' }, + consumer: { client: 'claude-code', arm: 'nested-consumer-value' }, }); } const filtered = evidenceReport(dir, { filters: { arm: 'natural' } }); From ddd22224d8da8976164c36b582daf05a73614bd0 Mon Sep 17 00:00:00 2001 From: Franklin Moormann Date: Sun, 9 Aug 2026 18:02:37 -0400 Subject: [PATCH 9/9] fix: deliver relevant session index --- hooks-core/adapter.mjs | 28 +++++++++- hooks-core/inject.mjs | 55 ++++++++++++++++++- .../hooks/token-optimizer/lib/adapter.mjs | 28 +++++++++- .../hooks/token-optimizer/lib/inject.mjs | 55 ++++++++++++++++++- integrations/codex/hooks/lib/adapter.mjs | 28 +++++++++- integrations/codex/hooks/lib/inject.mjs | 55 ++++++++++++++++++- .../codex/plugin/hooks/lib/adapter.mjs | 28 +++++++++- .../codex/plugin/hooks/lib/inject.mjs | 55 ++++++++++++++++++- .../copilot/.github/hooks/lib/adapter.mjs | 28 +++++++++- .../copilot/.github/hooks/lib/inject.mjs | 55 ++++++++++++++++++- integrations/cursor/hooks/lib/adapter.mjs | 28 +++++++++- integrations/cursor/hooks/lib/inject.mjs | 55 ++++++++++++++++++- integrations/gemini/hooks/lib/adapter.mjs | 28 +++++++++- integrations/gemini/hooks/lib/inject.mjs | 55 ++++++++++++++++++- integrations/kilo/hooks/lib/adapter.mjs | 28 +++++++++- integrations/kilo/hooks/lib/inject.mjs | 55 ++++++++++++++++++- integrations/opencode/hooks/lib/adapter.mjs | 28 +++++++++- integrations/opencode/hooks/lib/inject.mjs | 55 ++++++++++++++++++- integrations/qwen/hooks/lib/adapter.mjs | 28 +++++++++- integrations/qwen/hooks/lib/inject.mjs | 55 ++++++++++++++++++- integrations/windsurf/hooks/lib/adapter.mjs | 28 +++++++++- integrations/windsurf/hooks/lib/inject.mjs | 55 ++++++++++++++++++- plugin/hooks/lib/adapter.mjs | 28 +++++++++- plugin/hooks/lib/inject.mjs | 55 ++++++++++++++++++- plugin/hooks/session-start.mjs | 13 ++++- tests/hooks/injection.test.mjs | 26 ++++++++- tests/hooks/standing-rules.test.mjs | 35 ++++++++++++ 27 files changed, 1031 insertions(+), 39 deletions(-) diff --git a/hooks-core/adapter.mjs b/hooks-core/adapter.mjs index 55e4f644b..b643ab94b 100644 --- a/hooks-core/adapter.mjs +++ b/hooks-core/adapter.mjs @@ -50,6 +50,7 @@ import { forSharedCommand, forTouch, noteActClasses, + relevantFindingIdsForContext, sessionIndex, standingRules, } from './inject.mjs'; @@ -176,6 +177,24 @@ export function normalizeClientPayload(clientName, event, raw) { return raw; } +/** Extract only user/task text, never cwd, ids, or other metadata. */ +export function sessionTaskContext(raw = {}) { + return [ + raw.prompt, + raw.user_prompt, + raw.userPrompt, + raw.initial_prompt, + raw.initialPrompt, + raw.task_prompt, + raw.taskPrompt, + raw.task_description, + raw.taskDescription, + raw.task?.prompt, + raw.task?.text, + raw.message?.content, + ].filter((value) => typeof value === 'string' && value.trim()).join('\n'); +} + function emit(object) { process.stdout.write(JSON.stringify(object)); } @@ -437,7 +456,14 @@ export async function run(clientName, event) { const episode = episodeMeta({ client: clientName, raw }); const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); - const index = sessionIndex(dir, graph, { episode }); + const relevantFindingIds = relevantFindingIdsForContext( + graph, + sessionTaskContext(raw) + ); + const index = sessionIndex(dir, graph, { + episode, + relevantFindingIds, + }); if (index) parts.push(index); } catch { // Retrieval is optional context; the policy must still reach the model. diff --git a/hooks-core/inject.mjs b/hooks-core/inject.mjs index 7efeb7923..b2d686134 100644 --- a/hooks-core/inject.mjs +++ b/hooks-core/inject.mjs @@ -770,6 +770,57 @@ export function forSharedCommand( } +const RELEVANCE_STOP_WORDS = new Set([ + 'about', 'after', 'again', 'before', 'could', 'from', 'have', 'into', + 'just', 'need', 'that', 'their', 'there', 'these', 'this', 'those', + 'using', 'want', 'when', 'where', 'which', 'with', 'would', 'your', +]); + +function relevanceTerms(value) { + return [...new Set(String(value || '').toLowerCase() + .split(/[^a-z0-9_.:/-]+/) + .map((term) => term.replace(/^[_.:/-]+|[_.:/-]+$/g, '')) + .filter((term) => term.length >= 3 && !RELEVANCE_STOP_WORDS.has(term)))]; +} + +/** + * Selects finding keys only when a lifecycle payload contains actual task text. + * Session-start envelopes that carry only cwd/session metadata deliberately + * return no ids; their first file/command event uses the contextual injectors. + */ +export function relevantFindingIdsForContext(graph, context, { limit = 8 } = {}) { + const query = String(context || '').trim(); + const queryTerms = relevanceTerms(query); + if (queryTerms.length < 2) return []; + + return [...graph.nodes.values()] + .filter((finding) => + finding.kind === 'finding' + && finding.retired !== true + && typeof finding.key === 'string' + && typeof finding.claim === 'string') + .map((finding) => { + const content = [ + finding.key, + finding.claim, + finding.trigger, + ...(finding.anchors || []), + ].filter(Boolean).join(' '); + const terms = new Set(relevanceTerms(content)); + const matches = queryTerms.filter((term) => terms.has(term)).length; + return { finding, matches }; + }) + // Two independent terms keeps a generic word such as "test" or "file" + // from turning the bounded index into an unrelated project-wide dump. + .filter(({ matches }) => matches >= 2) + .sort((a, b) => + b.matches - a.matches + || (b.finding.confidence || 0.5) - (a.finding.confidence || 0.5) + || a.finding.key.localeCompare(b.finding.key)) + .slice(0, Math.max(0, limit)) + .map(({ finding }) => finding.key); +} + /** * The bounded SessionStart index: titles and ids only, never bodies. * @@ -780,8 +831,8 @@ export function forSharedCommand( export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); const relevant = new Set(relevantFindingIds); - // SessionStart has no task signal. Fail closed for situational findings until - // a client supplies ids selected from an actual prompt/tool relevance pass. + // Some SessionStart payloads have no task signal. Fail closed for situational + // findings until the adapter supplies ids selected from actual task text. // Universal human/pinned rules are delivered separately by `standingRules`. if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read diff --git a/integrations/cline/hooks/token-optimizer/lib/adapter.mjs b/integrations/cline/hooks/token-optimizer/lib/adapter.mjs index 0e079eca7..78a5cd60d 100644 --- a/integrations/cline/hooks/token-optimizer/lib/adapter.mjs +++ b/integrations/cline/hooks/token-optimizer/lib/adapter.mjs @@ -52,6 +52,7 @@ import { forSharedCommand, forTouch, noteActClasses, + relevantFindingIdsForContext, sessionIndex, standingRules, } from './inject.mjs'; @@ -178,6 +179,24 @@ export function normalizeClientPayload(clientName, event, raw) { return raw; } +/** Extract only user/task text, never cwd, ids, or other metadata. */ +export function sessionTaskContext(raw = {}) { + return [ + raw.prompt, + raw.user_prompt, + raw.userPrompt, + raw.initial_prompt, + raw.initialPrompt, + raw.task_prompt, + raw.taskPrompt, + raw.task_description, + raw.taskDescription, + raw.task?.prompt, + raw.task?.text, + raw.message?.content, + ].filter((value) => typeof value === 'string' && value.trim()).join('\n'); +} + function emit(object) { process.stdout.write(JSON.stringify(object)); } @@ -439,7 +458,14 @@ export async function run(clientName, event) { const episode = episodeMeta({ client: clientName, raw }); const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); - const index = sessionIndex(dir, graph, { episode }); + const relevantFindingIds = relevantFindingIdsForContext( + graph, + sessionTaskContext(raw) + ); + const index = sessionIndex(dir, graph, { + episode, + relevantFindingIds, + }); if (index) parts.push(index); } catch { // Retrieval is optional context; the policy must still reach the model. diff --git a/integrations/cline/hooks/token-optimizer/lib/inject.mjs b/integrations/cline/hooks/token-optimizer/lib/inject.mjs index 1ec5b4ce4..b00494a01 100644 --- a/integrations/cline/hooks/token-optimizer/lib/inject.mjs +++ b/integrations/cline/hooks/token-optimizer/lib/inject.mjs @@ -772,6 +772,57 @@ export function forSharedCommand( } +const RELEVANCE_STOP_WORDS = new Set([ + 'about', 'after', 'again', 'before', 'could', 'from', 'have', 'into', + 'just', 'need', 'that', 'their', 'there', 'these', 'this', 'those', + 'using', 'want', 'when', 'where', 'which', 'with', 'would', 'your', +]); + +function relevanceTerms(value) { + return [...new Set(String(value || '').toLowerCase() + .split(/[^a-z0-9_.:/-]+/) + .map((term) => term.replace(/^[_.:/-]+|[_.:/-]+$/g, '')) + .filter((term) => term.length >= 3 && !RELEVANCE_STOP_WORDS.has(term)))]; +} + +/** + * Selects finding keys only when a lifecycle payload contains actual task text. + * Session-start envelopes that carry only cwd/session metadata deliberately + * return no ids; their first file/command event uses the contextual injectors. + */ +export function relevantFindingIdsForContext(graph, context, { limit = 8 } = {}) { + const query = String(context || '').trim(); + const queryTerms = relevanceTerms(query); + if (queryTerms.length < 2) return []; + + return [...graph.nodes.values()] + .filter((finding) => + finding.kind === 'finding' + && finding.retired !== true + && typeof finding.key === 'string' + && typeof finding.claim === 'string') + .map((finding) => { + const content = [ + finding.key, + finding.claim, + finding.trigger, + ...(finding.anchors || []), + ].filter(Boolean).join(' '); + const terms = new Set(relevanceTerms(content)); + const matches = queryTerms.filter((term) => terms.has(term)).length; + return { finding, matches }; + }) + // Two independent terms keeps a generic word such as "test" or "file" + // from turning the bounded index into an unrelated project-wide dump. + .filter(({ matches }) => matches >= 2) + .sort((a, b) => + b.matches - a.matches + || (b.finding.confidence || 0.5) - (a.finding.confidence || 0.5) + || a.finding.key.localeCompare(b.finding.key)) + .slice(0, Math.max(0, limit)) + .map(({ finding }) => finding.key); +} + /** * The bounded SessionStart index: titles and ids only, never bodies. * @@ -782,8 +833,8 @@ export function forSharedCommand( export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); const relevant = new Set(relevantFindingIds); - // SessionStart has no task signal. Fail closed for situational findings until - // a client supplies ids selected from an actual prompt/tool relevance pass. + // Some SessionStart payloads have no task signal. Fail closed for situational + // findings until the adapter supplies ids selected from actual task text. // Universal human/pinned rules are delivered separately by `standingRules`. if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read diff --git a/integrations/codex/hooks/lib/adapter.mjs b/integrations/codex/hooks/lib/adapter.mjs index 0e079eca7..78a5cd60d 100644 --- a/integrations/codex/hooks/lib/adapter.mjs +++ b/integrations/codex/hooks/lib/adapter.mjs @@ -52,6 +52,7 @@ import { forSharedCommand, forTouch, noteActClasses, + relevantFindingIdsForContext, sessionIndex, standingRules, } from './inject.mjs'; @@ -178,6 +179,24 @@ export function normalizeClientPayload(clientName, event, raw) { return raw; } +/** Extract only user/task text, never cwd, ids, or other metadata. */ +export function sessionTaskContext(raw = {}) { + return [ + raw.prompt, + raw.user_prompt, + raw.userPrompt, + raw.initial_prompt, + raw.initialPrompt, + raw.task_prompt, + raw.taskPrompt, + raw.task_description, + raw.taskDescription, + raw.task?.prompt, + raw.task?.text, + raw.message?.content, + ].filter((value) => typeof value === 'string' && value.trim()).join('\n'); +} + function emit(object) { process.stdout.write(JSON.stringify(object)); } @@ -439,7 +458,14 @@ export async function run(clientName, event) { const episode = episodeMeta({ client: clientName, raw }); const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); - const index = sessionIndex(dir, graph, { episode }); + const relevantFindingIds = relevantFindingIdsForContext( + graph, + sessionTaskContext(raw) + ); + const index = sessionIndex(dir, graph, { + episode, + relevantFindingIds, + }); if (index) parts.push(index); } catch { // Retrieval is optional context; the policy must still reach the model. diff --git a/integrations/codex/hooks/lib/inject.mjs b/integrations/codex/hooks/lib/inject.mjs index 1ec5b4ce4..b00494a01 100644 --- a/integrations/codex/hooks/lib/inject.mjs +++ b/integrations/codex/hooks/lib/inject.mjs @@ -772,6 +772,57 @@ export function forSharedCommand( } +const RELEVANCE_STOP_WORDS = new Set([ + 'about', 'after', 'again', 'before', 'could', 'from', 'have', 'into', + 'just', 'need', 'that', 'their', 'there', 'these', 'this', 'those', + 'using', 'want', 'when', 'where', 'which', 'with', 'would', 'your', +]); + +function relevanceTerms(value) { + return [...new Set(String(value || '').toLowerCase() + .split(/[^a-z0-9_.:/-]+/) + .map((term) => term.replace(/^[_.:/-]+|[_.:/-]+$/g, '')) + .filter((term) => term.length >= 3 && !RELEVANCE_STOP_WORDS.has(term)))]; +} + +/** + * Selects finding keys only when a lifecycle payload contains actual task text. + * Session-start envelopes that carry only cwd/session metadata deliberately + * return no ids; their first file/command event uses the contextual injectors. + */ +export function relevantFindingIdsForContext(graph, context, { limit = 8 } = {}) { + const query = String(context || '').trim(); + const queryTerms = relevanceTerms(query); + if (queryTerms.length < 2) return []; + + return [...graph.nodes.values()] + .filter((finding) => + finding.kind === 'finding' + && finding.retired !== true + && typeof finding.key === 'string' + && typeof finding.claim === 'string') + .map((finding) => { + const content = [ + finding.key, + finding.claim, + finding.trigger, + ...(finding.anchors || []), + ].filter(Boolean).join(' '); + const terms = new Set(relevanceTerms(content)); + const matches = queryTerms.filter((term) => terms.has(term)).length; + return { finding, matches }; + }) + // Two independent terms keeps a generic word such as "test" or "file" + // from turning the bounded index into an unrelated project-wide dump. + .filter(({ matches }) => matches >= 2) + .sort((a, b) => + b.matches - a.matches + || (b.finding.confidence || 0.5) - (a.finding.confidence || 0.5) + || a.finding.key.localeCompare(b.finding.key)) + .slice(0, Math.max(0, limit)) + .map(({ finding }) => finding.key); +} + /** * The bounded SessionStart index: titles and ids only, never bodies. * @@ -782,8 +833,8 @@ export function forSharedCommand( export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); const relevant = new Set(relevantFindingIds); - // SessionStart has no task signal. Fail closed for situational findings until - // a client supplies ids selected from an actual prompt/tool relevance pass. + // Some SessionStart payloads have no task signal. Fail closed for situational + // findings until the adapter supplies ids selected from actual task text. // Universal human/pinned rules are delivered separately by `standingRules`. if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read diff --git a/integrations/codex/plugin/hooks/lib/adapter.mjs b/integrations/codex/plugin/hooks/lib/adapter.mjs index 0e079eca7..78a5cd60d 100644 --- a/integrations/codex/plugin/hooks/lib/adapter.mjs +++ b/integrations/codex/plugin/hooks/lib/adapter.mjs @@ -52,6 +52,7 @@ import { forSharedCommand, forTouch, noteActClasses, + relevantFindingIdsForContext, sessionIndex, standingRules, } from './inject.mjs'; @@ -178,6 +179,24 @@ export function normalizeClientPayload(clientName, event, raw) { return raw; } +/** Extract only user/task text, never cwd, ids, or other metadata. */ +export function sessionTaskContext(raw = {}) { + return [ + raw.prompt, + raw.user_prompt, + raw.userPrompt, + raw.initial_prompt, + raw.initialPrompt, + raw.task_prompt, + raw.taskPrompt, + raw.task_description, + raw.taskDescription, + raw.task?.prompt, + raw.task?.text, + raw.message?.content, + ].filter((value) => typeof value === 'string' && value.trim()).join('\n'); +} + function emit(object) { process.stdout.write(JSON.stringify(object)); } @@ -439,7 +458,14 @@ export async function run(clientName, event) { const episode = episodeMeta({ client: clientName, raw }); const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); - const index = sessionIndex(dir, graph, { episode }); + const relevantFindingIds = relevantFindingIdsForContext( + graph, + sessionTaskContext(raw) + ); + const index = sessionIndex(dir, graph, { + episode, + relevantFindingIds, + }); if (index) parts.push(index); } catch { // Retrieval is optional context; the policy must still reach the model. diff --git a/integrations/codex/plugin/hooks/lib/inject.mjs b/integrations/codex/plugin/hooks/lib/inject.mjs index 1ec5b4ce4..b00494a01 100644 --- a/integrations/codex/plugin/hooks/lib/inject.mjs +++ b/integrations/codex/plugin/hooks/lib/inject.mjs @@ -772,6 +772,57 @@ export function forSharedCommand( } +const RELEVANCE_STOP_WORDS = new Set([ + 'about', 'after', 'again', 'before', 'could', 'from', 'have', 'into', + 'just', 'need', 'that', 'their', 'there', 'these', 'this', 'those', + 'using', 'want', 'when', 'where', 'which', 'with', 'would', 'your', +]); + +function relevanceTerms(value) { + return [...new Set(String(value || '').toLowerCase() + .split(/[^a-z0-9_.:/-]+/) + .map((term) => term.replace(/^[_.:/-]+|[_.:/-]+$/g, '')) + .filter((term) => term.length >= 3 && !RELEVANCE_STOP_WORDS.has(term)))]; +} + +/** + * Selects finding keys only when a lifecycle payload contains actual task text. + * Session-start envelopes that carry only cwd/session metadata deliberately + * return no ids; their first file/command event uses the contextual injectors. + */ +export function relevantFindingIdsForContext(graph, context, { limit = 8 } = {}) { + const query = String(context || '').trim(); + const queryTerms = relevanceTerms(query); + if (queryTerms.length < 2) return []; + + return [...graph.nodes.values()] + .filter((finding) => + finding.kind === 'finding' + && finding.retired !== true + && typeof finding.key === 'string' + && typeof finding.claim === 'string') + .map((finding) => { + const content = [ + finding.key, + finding.claim, + finding.trigger, + ...(finding.anchors || []), + ].filter(Boolean).join(' '); + const terms = new Set(relevanceTerms(content)); + const matches = queryTerms.filter((term) => terms.has(term)).length; + return { finding, matches }; + }) + // Two independent terms keeps a generic word such as "test" or "file" + // from turning the bounded index into an unrelated project-wide dump. + .filter(({ matches }) => matches >= 2) + .sort((a, b) => + b.matches - a.matches + || (b.finding.confidence || 0.5) - (a.finding.confidence || 0.5) + || a.finding.key.localeCompare(b.finding.key)) + .slice(0, Math.max(0, limit)) + .map(({ finding }) => finding.key); +} + /** * The bounded SessionStart index: titles and ids only, never bodies. * @@ -782,8 +833,8 @@ export function forSharedCommand( export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); const relevant = new Set(relevantFindingIds); - // SessionStart has no task signal. Fail closed for situational findings until - // a client supplies ids selected from an actual prompt/tool relevance pass. + // Some SessionStart payloads have no task signal. Fail closed for situational + // findings until the adapter supplies ids selected from actual task text. // Universal human/pinned rules are delivered separately by `standingRules`. if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read diff --git a/integrations/copilot/.github/hooks/lib/adapter.mjs b/integrations/copilot/.github/hooks/lib/adapter.mjs index 0e079eca7..78a5cd60d 100644 --- a/integrations/copilot/.github/hooks/lib/adapter.mjs +++ b/integrations/copilot/.github/hooks/lib/adapter.mjs @@ -52,6 +52,7 @@ import { forSharedCommand, forTouch, noteActClasses, + relevantFindingIdsForContext, sessionIndex, standingRules, } from './inject.mjs'; @@ -178,6 +179,24 @@ export function normalizeClientPayload(clientName, event, raw) { return raw; } +/** Extract only user/task text, never cwd, ids, or other metadata. */ +export function sessionTaskContext(raw = {}) { + return [ + raw.prompt, + raw.user_prompt, + raw.userPrompt, + raw.initial_prompt, + raw.initialPrompt, + raw.task_prompt, + raw.taskPrompt, + raw.task_description, + raw.taskDescription, + raw.task?.prompt, + raw.task?.text, + raw.message?.content, + ].filter((value) => typeof value === 'string' && value.trim()).join('\n'); +} + function emit(object) { process.stdout.write(JSON.stringify(object)); } @@ -439,7 +458,14 @@ export async function run(clientName, event) { const episode = episodeMeta({ client: clientName, raw }); const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); - const index = sessionIndex(dir, graph, { episode }); + const relevantFindingIds = relevantFindingIdsForContext( + graph, + sessionTaskContext(raw) + ); + const index = sessionIndex(dir, graph, { + episode, + relevantFindingIds, + }); if (index) parts.push(index); } catch { // Retrieval is optional context; the policy must still reach the model. diff --git a/integrations/copilot/.github/hooks/lib/inject.mjs b/integrations/copilot/.github/hooks/lib/inject.mjs index 1ec5b4ce4..b00494a01 100644 --- a/integrations/copilot/.github/hooks/lib/inject.mjs +++ b/integrations/copilot/.github/hooks/lib/inject.mjs @@ -772,6 +772,57 @@ export function forSharedCommand( } +const RELEVANCE_STOP_WORDS = new Set([ + 'about', 'after', 'again', 'before', 'could', 'from', 'have', 'into', + 'just', 'need', 'that', 'their', 'there', 'these', 'this', 'those', + 'using', 'want', 'when', 'where', 'which', 'with', 'would', 'your', +]); + +function relevanceTerms(value) { + return [...new Set(String(value || '').toLowerCase() + .split(/[^a-z0-9_.:/-]+/) + .map((term) => term.replace(/^[_.:/-]+|[_.:/-]+$/g, '')) + .filter((term) => term.length >= 3 && !RELEVANCE_STOP_WORDS.has(term)))]; +} + +/** + * Selects finding keys only when a lifecycle payload contains actual task text. + * Session-start envelopes that carry only cwd/session metadata deliberately + * return no ids; their first file/command event uses the contextual injectors. + */ +export function relevantFindingIdsForContext(graph, context, { limit = 8 } = {}) { + const query = String(context || '').trim(); + const queryTerms = relevanceTerms(query); + if (queryTerms.length < 2) return []; + + return [...graph.nodes.values()] + .filter((finding) => + finding.kind === 'finding' + && finding.retired !== true + && typeof finding.key === 'string' + && typeof finding.claim === 'string') + .map((finding) => { + const content = [ + finding.key, + finding.claim, + finding.trigger, + ...(finding.anchors || []), + ].filter(Boolean).join(' '); + const terms = new Set(relevanceTerms(content)); + const matches = queryTerms.filter((term) => terms.has(term)).length; + return { finding, matches }; + }) + // Two independent terms keeps a generic word such as "test" or "file" + // from turning the bounded index into an unrelated project-wide dump. + .filter(({ matches }) => matches >= 2) + .sort((a, b) => + b.matches - a.matches + || (b.finding.confidence || 0.5) - (a.finding.confidence || 0.5) + || a.finding.key.localeCompare(b.finding.key)) + .slice(0, Math.max(0, limit)) + .map(({ finding }) => finding.key); +} + /** * The bounded SessionStart index: titles and ids only, never bodies. * @@ -782,8 +833,8 @@ export function forSharedCommand( export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); const relevant = new Set(relevantFindingIds); - // SessionStart has no task signal. Fail closed for situational findings until - // a client supplies ids selected from an actual prompt/tool relevance pass. + // Some SessionStart payloads have no task signal. Fail closed for situational + // findings until the adapter supplies ids selected from actual task text. // Universal human/pinned rules are delivered separately by `standingRules`. if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read diff --git a/integrations/cursor/hooks/lib/adapter.mjs b/integrations/cursor/hooks/lib/adapter.mjs index 0e079eca7..78a5cd60d 100644 --- a/integrations/cursor/hooks/lib/adapter.mjs +++ b/integrations/cursor/hooks/lib/adapter.mjs @@ -52,6 +52,7 @@ import { forSharedCommand, forTouch, noteActClasses, + relevantFindingIdsForContext, sessionIndex, standingRules, } from './inject.mjs'; @@ -178,6 +179,24 @@ export function normalizeClientPayload(clientName, event, raw) { return raw; } +/** Extract only user/task text, never cwd, ids, or other metadata. */ +export function sessionTaskContext(raw = {}) { + return [ + raw.prompt, + raw.user_prompt, + raw.userPrompt, + raw.initial_prompt, + raw.initialPrompt, + raw.task_prompt, + raw.taskPrompt, + raw.task_description, + raw.taskDescription, + raw.task?.prompt, + raw.task?.text, + raw.message?.content, + ].filter((value) => typeof value === 'string' && value.trim()).join('\n'); +} + function emit(object) { process.stdout.write(JSON.stringify(object)); } @@ -439,7 +458,14 @@ export async function run(clientName, event) { const episode = episodeMeta({ client: clientName, raw }); const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); - const index = sessionIndex(dir, graph, { episode }); + const relevantFindingIds = relevantFindingIdsForContext( + graph, + sessionTaskContext(raw) + ); + const index = sessionIndex(dir, graph, { + episode, + relevantFindingIds, + }); if (index) parts.push(index); } catch { // Retrieval is optional context; the policy must still reach the model. diff --git a/integrations/cursor/hooks/lib/inject.mjs b/integrations/cursor/hooks/lib/inject.mjs index 1ec5b4ce4..b00494a01 100644 --- a/integrations/cursor/hooks/lib/inject.mjs +++ b/integrations/cursor/hooks/lib/inject.mjs @@ -772,6 +772,57 @@ export function forSharedCommand( } +const RELEVANCE_STOP_WORDS = new Set([ + 'about', 'after', 'again', 'before', 'could', 'from', 'have', 'into', + 'just', 'need', 'that', 'their', 'there', 'these', 'this', 'those', + 'using', 'want', 'when', 'where', 'which', 'with', 'would', 'your', +]); + +function relevanceTerms(value) { + return [...new Set(String(value || '').toLowerCase() + .split(/[^a-z0-9_.:/-]+/) + .map((term) => term.replace(/^[_.:/-]+|[_.:/-]+$/g, '')) + .filter((term) => term.length >= 3 && !RELEVANCE_STOP_WORDS.has(term)))]; +} + +/** + * Selects finding keys only when a lifecycle payload contains actual task text. + * Session-start envelopes that carry only cwd/session metadata deliberately + * return no ids; their first file/command event uses the contextual injectors. + */ +export function relevantFindingIdsForContext(graph, context, { limit = 8 } = {}) { + const query = String(context || '').trim(); + const queryTerms = relevanceTerms(query); + if (queryTerms.length < 2) return []; + + return [...graph.nodes.values()] + .filter((finding) => + finding.kind === 'finding' + && finding.retired !== true + && typeof finding.key === 'string' + && typeof finding.claim === 'string') + .map((finding) => { + const content = [ + finding.key, + finding.claim, + finding.trigger, + ...(finding.anchors || []), + ].filter(Boolean).join(' '); + const terms = new Set(relevanceTerms(content)); + const matches = queryTerms.filter((term) => terms.has(term)).length; + return { finding, matches }; + }) + // Two independent terms keeps a generic word such as "test" or "file" + // from turning the bounded index into an unrelated project-wide dump. + .filter(({ matches }) => matches >= 2) + .sort((a, b) => + b.matches - a.matches + || (b.finding.confidence || 0.5) - (a.finding.confidence || 0.5) + || a.finding.key.localeCompare(b.finding.key)) + .slice(0, Math.max(0, limit)) + .map(({ finding }) => finding.key); +} + /** * The bounded SessionStart index: titles and ids only, never bodies. * @@ -782,8 +833,8 @@ export function forSharedCommand( export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); const relevant = new Set(relevantFindingIds); - // SessionStart has no task signal. Fail closed for situational findings until - // a client supplies ids selected from an actual prompt/tool relevance pass. + // Some SessionStart payloads have no task signal. Fail closed for situational + // findings until the adapter supplies ids selected from actual task text. // Universal human/pinned rules are delivered separately by `standingRules`. if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read diff --git a/integrations/gemini/hooks/lib/adapter.mjs b/integrations/gemini/hooks/lib/adapter.mjs index 0e079eca7..78a5cd60d 100644 --- a/integrations/gemini/hooks/lib/adapter.mjs +++ b/integrations/gemini/hooks/lib/adapter.mjs @@ -52,6 +52,7 @@ import { forSharedCommand, forTouch, noteActClasses, + relevantFindingIdsForContext, sessionIndex, standingRules, } from './inject.mjs'; @@ -178,6 +179,24 @@ export function normalizeClientPayload(clientName, event, raw) { return raw; } +/** Extract only user/task text, never cwd, ids, or other metadata. */ +export function sessionTaskContext(raw = {}) { + return [ + raw.prompt, + raw.user_prompt, + raw.userPrompt, + raw.initial_prompt, + raw.initialPrompt, + raw.task_prompt, + raw.taskPrompt, + raw.task_description, + raw.taskDescription, + raw.task?.prompt, + raw.task?.text, + raw.message?.content, + ].filter((value) => typeof value === 'string' && value.trim()).join('\n'); +} + function emit(object) { process.stdout.write(JSON.stringify(object)); } @@ -439,7 +458,14 @@ export async function run(clientName, event) { const episode = episodeMeta({ client: clientName, raw }); const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); - const index = sessionIndex(dir, graph, { episode }); + const relevantFindingIds = relevantFindingIdsForContext( + graph, + sessionTaskContext(raw) + ); + const index = sessionIndex(dir, graph, { + episode, + relevantFindingIds, + }); if (index) parts.push(index); } catch { // Retrieval is optional context; the policy must still reach the model. diff --git a/integrations/gemini/hooks/lib/inject.mjs b/integrations/gemini/hooks/lib/inject.mjs index 1ec5b4ce4..b00494a01 100644 --- a/integrations/gemini/hooks/lib/inject.mjs +++ b/integrations/gemini/hooks/lib/inject.mjs @@ -772,6 +772,57 @@ export function forSharedCommand( } +const RELEVANCE_STOP_WORDS = new Set([ + 'about', 'after', 'again', 'before', 'could', 'from', 'have', 'into', + 'just', 'need', 'that', 'their', 'there', 'these', 'this', 'those', + 'using', 'want', 'when', 'where', 'which', 'with', 'would', 'your', +]); + +function relevanceTerms(value) { + return [...new Set(String(value || '').toLowerCase() + .split(/[^a-z0-9_.:/-]+/) + .map((term) => term.replace(/^[_.:/-]+|[_.:/-]+$/g, '')) + .filter((term) => term.length >= 3 && !RELEVANCE_STOP_WORDS.has(term)))]; +} + +/** + * Selects finding keys only when a lifecycle payload contains actual task text. + * Session-start envelopes that carry only cwd/session metadata deliberately + * return no ids; their first file/command event uses the contextual injectors. + */ +export function relevantFindingIdsForContext(graph, context, { limit = 8 } = {}) { + const query = String(context || '').trim(); + const queryTerms = relevanceTerms(query); + if (queryTerms.length < 2) return []; + + return [...graph.nodes.values()] + .filter((finding) => + finding.kind === 'finding' + && finding.retired !== true + && typeof finding.key === 'string' + && typeof finding.claim === 'string') + .map((finding) => { + const content = [ + finding.key, + finding.claim, + finding.trigger, + ...(finding.anchors || []), + ].filter(Boolean).join(' '); + const terms = new Set(relevanceTerms(content)); + const matches = queryTerms.filter((term) => terms.has(term)).length; + return { finding, matches }; + }) + // Two independent terms keeps a generic word such as "test" or "file" + // from turning the bounded index into an unrelated project-wide dump. + .filter(({ matches }) => matches >= 2) + .sort((a, b) => + b.matches - a.matches + || (b.finding.confidence || 0.5) - (a.finding.confidence || 0.5) + || a.finding.key.localeCompare(b.finding.key)) + .slice(0, Math.max(0, limit)) + .map(({ finding }) => finding.key); +} + /** * The bounded SessionStart index: titles and ids only, never bodies. * @@ -782,8 +833,8 @@ export function forSharedCommand( export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); const relevant = new Set(relevantFindingIds); - // SessionStart has no task signal. Fail closed for situational findings until - // a client supplies ids selected from an actual prompt/tool relevance pass. + // Some SessionStart payloads have no task signal. Fail closed for situational + // findings until the adapter supplies ids selected from actual task text. // Universal human/pinned rules are delivered separately by `standingRules`. if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read diff --git a/integrations/kilo/hooks/lib/adapter.mjs b/integrations/kilo/hooks/lib/adapter.mjs index 0e079eca7..78a5cd60d 100644 --- a/integrations/kilo/hooks/lib/adapter.mjs +++ b/integrations/kilo/hooks/lib/adapter.mjs @@ -52,6 +52,7 @@ import { forSharedCommand, forTouch, noteActClasses, + relevantFindingIdsForContext, sessionIndex, standingRules, } from './inject.mjs'; @@ -178,6 +179,24 @@ export function normalizeClientPayload(clientName, event, raw) { return raw; } +/** Extract only user/task text, never cwd, ids, or other metadata. */ +export function sessionTaskContext(raw = {}) { + return [ + raw.prompt, + raw.user_prompt, + raw.userPrompt, + raw.initial_prompt, + raw.initialPrompt, + raw.task_prompt, + raw.taskPrompt, + raw.task_description, + raw.taskDescription, + raw.task?.prompt, + raw.task?.text, + raw.message?.content, + ].filter((value) => typeof value === 'string' && value.trim()).join('\n'); +} + function emit(object) { process.stdout.write(JSON.stringify(object)); } @@ -439,7 +458,14 @@ export async function run(clientName, event) { const episode = episodeMeta({ client: clientName, raw }); const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); - const index = sessionIndex(dir, graph, { episode }); + const relevantFindingIds = relevantFindingIdsForContext( + graph, + sessionTaskContext(raw) + ); + const index = sessionIndex(dir, graph, { + episode, + relevantFindingIds, + }); if (index) parts.push(index); } catch { // Retrieval is optional context; the policy must still reach the model. diff --git a/integrations/kilo/hooks/lib/inject.mjs b/integrations/kilo/hooks/lib/inject.mjs index 1ec5b4ce4..b00494a01 100644 --- a/integrations/kilo/hooks/lib/inject.mjs +++ b/integrations/kilo/hooks/lib/inject.mjs @@ -772,6 +772,57 @@ export function forSharedCommand( } +const RELEVANCE_STOP_WORDS = new Set([ + 'about', 'after', 'again', 'before', 'could', 'from', 'have', 'into', + 'just', 'need', 'that', 'their', 'there', 'these', 'this', 'those', + 'using', 'want', 'when', 'where', 'which', 'with', 'would', 'your', +]); + +function relevanceTerms(value) { + return [...new Set(String(value || '').toLowerCase() + .split(/[^a-z0-9_.:/-]+/) + .map((term) => term.replace(/^[_.:/-]+|[_.:/-]+$/g, '')) + .filter((term) => term.length >= 3 && !RELEVANCE_STOP_WORDS.has(term)))]; +} + +/** + * Selects finding keys only when a lifecycle payload contains actual task text. + * Session-start envelopes that carry only cwd/session metadata deliberately + * return no ids; their first file/command event uses the contextual injectors. + */ +export function relevantFindingIdsForContext(graph, context, { limit = 8 } = {}) { + const query = String(context || '').trim(); + const queryTerms = relevanceTerms(query); + if (queryTerms.length < 2) return []; + + return [...graph.nodes.values()] + .filter((finding) => + finding.kind === 'finding' + && finding.retired !== true + && typeof finding.key === 'string' + && typeof finding.claim === 'string') + .map((finding) => { + const content = [ + finding.key, + finding.claim, + finding.trigger, + ...(finding.anchors || []), + ].filter(Boolean).join(' '); + const terms = new Set(relevanceTerms(content)); + const matches = queryTerms.filter((term) => terms.has(term)).length; + return { finding, matches }; + }) + // Two independent terms keeps a generic word such as "test" or "file" + // from turning the bounded index into an unrelated project-wide dump. + .filter(({ matches }) => matches >= 2) + .sort((a, b) => + b.matches - a.matches + || (b.finding.confidence || 0.5) - (a.finding.confidence || 0.5) + || a.finding.key.localeCompare(b.finding.key)) + .slice(0, Math.max(0, limit)) + .map(({ finding }) => finding.key); +} + /** * The bounded SessionStart index: titles and ids only, never bodies. * @@ -782,8 +833,8 @@ export function forSharedCommand( export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); const relevant = new Set(relevantFindingIds); - // SessionStart has no task signal. Fail closed for situational findings until - // a client supplies ids selected from an actual prompt/tool relevance pass. + // Some SessionStart payloads have no task signal. Fail closed for situational + // findings until the adapter supplies ids selected from actual task text. // Universal human/pinned rules are delivered separately by `standingRules`. if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read diff --git a/integrations/opencode/hooks/lib/adapter.mjs b/integrations/opencode/hooks/lib/adapter.mjs index 0e079eca7..78a5cd60d 100644 --- a/integrations/opencode/hooks/lib/adapter.mjs +++ b/integrations/opencode/hooks/lib/adapter.mjs @@ -52,6 +52,7 @@ import { forSharedCommand, forTouch, noteActClasses, + relevantFindingIdsForContext, sessionIndex, standingRules, } from './inject.mjs'; @@ -178,6 +179,24 @@ export function normalizeClientPayload(clientName, event, raw) { return raw; } +/** Extract only user/task text, never cwd, ids, or other metadata. */ +export function sessionTaskContext(raw = {}) { + return [ + raw.prompt, + raw.user_prompt, + raw.userPrompt, + raw.initial_prompt, + raw.initialPrompt, + raw.task_prompt, + raw.taskPrompt, + raw.task_description, + raw.taskDescription, + raw.task?.prompt, + raw.task?.text, + raw.message?.content, + ].filter((value) => typeof value === 'string' && value.trim()).join('\n'); +} + function emit(object) { process.stdout.write(JSON.stringify(object)); } @@ -439,7 +458,14 @@ export async function run(clientName, event) { const episode = episodeMeta({ client: clientName, raw }); const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); - const index = sessionIndex(dir, graph, { episode }); + const relevantFindingIds = relevantFindingIdsForContext( + graph, + sessionTaskContext(raw) + ); + const index = sessionIndex(dir, graph, { + episode, + relevantFindingIds, + }); if (index) parts.push(index); } catch { // Retrieval is optional context; the policy must still reach the model. diff --git a/integrations/opencode/hooks/lib/inject.mjs b/integrations/opencode/hooks/lib/inject.mjs index 1ec5b4ce4..b00494a01 100644 --- a/integrations/opencode/hooks/lib/inject.mjs +++ b/integrations/opencode/hooks/lib/inject.mjs @@ -772,6 +772,57 @@ export function forSharedCommand( } +const RELEVANCE_STOP_WORDS = new Set([ + 'about', 'after', 'again', 'before', 'could', 'from', 'have', 'into', + 'just', 'need', 'that', 'their', 'there', 'these', 'this', 'those', + 'using', 'want', 'when', 'where', 'which', 'with', 'would', 'your', +]); + +function relevanceTerms(value) { + return [...new Set(String(value || '').toLowerCase() + .split(/[^a-z0-9_.:/-]+/) + .map((term) => term.replace(/^[_.:/-]+|[_.:/-]+$/g, '')) + .filter((term) => term.length >= 3 && !RELEVANCE_STOP_WORDS.has(term)))]; +} + +/** + * Selects finding keys only when a lifecycle payload contains actual task text. + * Session-start envelopes that carry only cwd/session metadata deliberately + * return no ids; their first file/command event uses the contextual injectors. + */ +export function relevantFindingIdsForContext(graph, context, { limit = 8 } = {}) { + const query = String(context || '').trim(); + const queryTerms = relevanceTerms(query); + if (queryTerms.length < 2) return []; + + return [...graph.nodes.values()] + .filter((finding) => + finding.kind === 'finding' + && finding.retired !== true + && typeof finding.key === 'string' + && typeof finding.claim === 'string') + .map((finding) => { + const content = [ + finding.key, + finding.claim, + finding.trigger, + ...(finding.anchors || []), + ].filter(Boolean).join(' '); + const terms = new Set(relevanceTerms(content)); + const matches = queryTerms.filter((term) => terms.has(term)).length; + return { finding, matches }; + }) + // Two independent terms keeps a generic word such as "test" or "file" + // from turning the bounded index into an unrelated project-wide dump. + .filter(({ matches }) => matches >= 2) + .sort((a, b) => + b.matches - a.matches + || (b.finding.confidence || 0.5) - (a.finding.confidence || 0.5) + || a.finding.key.localeCompare(b.finding.key)) + .slice(0, Math.max(0, limit)) + .map(({ finding }) => finding.key); +} + /** * The bounded SessionStart index: titles and ids only, never bodies. * @@ -782,8 +833,8 @@ export function forSharedCommand( export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); const relevant = new Set(relevantFindingIds); - // SessionStart has no task signal. Fail closed for situational findings until - // a client supplies ids selected from an actual prompt/tool relevance pass. + // Some SessionStart payloads have no task signal. Fail closed for situational + // findings until the adapter supplies ids selected from actual task text. // Universal human/pinned rules are delivered separately by `standingRules`. if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read diff --git a/integrations/qwen/hooks/lib/adapter.mjs b/integrations/qwen/hooks/lib/adapter.mjs index 0e079eca7..78a5cd60d 100644 --- a/integrations/qwen/hooks/lib/adapter.mjs +++ b/integrations/qwen/hooks/lib/adapter.mjs @@ -52,6 +52,7 @@ import { forSharedCommand, forTouch, noteActClasses, + relevantFindingIdsForContext, sessionIndex, standingRules, } from './inject.mjs'; @@ -178,6 +179,24 @@ export function normalizeClientPayload(clientName, event, raw) { return raw; } +/** Extract only user/task text, never cwd, ids, or other metadata. */ +export function sessionTaskContext(raw = {}) { + return [ + raw.prompt, + raw.user_prompt, + raw.userPrompt, + raw.initial_prompt, + raw.initialPrompt, + raw.task_prompt, + raw.taskPrompt, + raw.task_description, + raw.taskDescription, + raw.task?.prompt, + raw.task?.text, + raw.message?.content, + ].filter((value) => typeof value === 'string' && value.trim()).join('\n'); +} + function emit(object) { process.stdout.write(JSON.stringify(object)); } @@ -439,7 +458,14 @@ export async function run(clientName, event) { const episode = episodeMeta({ client: clientName, raw }); const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); - const index = sessionIndex(dir, graph, { episode }); + const relevantFindingIds = relevantFindingIdsForContext( + graph, + sessionTaskContext(raw) + ); + const index = sessionIndex(dir, graph, { + episode, + relevantFindingIds, + }); if (index) parts.push(index); } catch { // Retrieval is optional context; the policy must still reach the model. diff --git a/integrations/qwen/hooks/lib/inject.mjs b/integrations/qwen/hooks/lib/inject.mjs index 1ec5b4ce4..b00494a01 100644 --- a/integrations/qwen/hooks/lib/inject.mjs +++ b/integrations/qwen/hooks/lib/inject.mjs @@ -772,6 +772,57 @@ export function forSharedCommand( } +const RELEVANCE_STOP_WORDS = new Set([ + 'about', 'after', 'again', 'before', 'could', 'from', 'have', 'into', + 'just', 'need', 'that', 'their', 'there', 'these', 'this', 'those', + 'using', 'want', 'when', 'where', 'which', 'with', 'would', 'your', +]); + +function relevanceTerms(value) { + return [...new Set(String(value || '').toLowerCase() + .split(/[^a-z0-9_.:/-]+/) + .map((term) => term.replace(/^[_.:/-]+|[_.:/-]+$/g, '')) + .filter((term) => term.length >= 3 && !RELEVANCE_STOP_WORDS.has(term)))]; +} + +/** + * Selects finding keys only when a lifecycle payload contains actual task text. + * Session-start envelopes that carry only cwd/session metadata deliberately + * return no ids; their first file/command event uses the contextual injectors. + */ +export function relevantFindingIdsForContext(graph, context, { limit = 8 } = {}) { + const query = String(context || '').trim(); + const queryTerms = relevanceTerms(query); + if (queryTerms.length < 2) return []; + + return [...graph.nodes.values()] + .filter((finding) => + finding.kind === 'finding' + && finding.retired !== true + && typeof finding.key === 'string' + && typeof finding.claim === 'string') + .map((finding) => { + const content = [ + finding.key, + finding.claim, + finding.trigger, + ...(finding.anchors || []), + ].filter(Boolean).join(' '); + const terms = new Set(relevanceTerms(content)); + const matches = queryTerms.filter((term) => terms.has(term)).length; + return { finding, matches }; + }) + // Two independent terms keeps a generic word such as "test" or "file" + // from turning the bounded index into an unrelated project-wide dump. + .filter(({ matches }) => matches >= 2) + .sort((a, b) => + b.matches - a.matches + || (b.finding.confidence || 0.5) - (a.finding.confidence || 0.5) + || a.finding.key.localeCompare(b.finding.key)) + .slice(0, Math.max(0, limit)) + .map(({ finding }) => finding.key); +} + /** * The bounded SessionStart index: titles and ids only, never bodies. * @@ -782,8 +833,8 @@ export function forSharedCommand( export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); const relevant = new Set(relevantFindingIds); - // SessionStart has no task signal. Fail closed for situational findings until - // a client supplies ids selected from an actual prompt/tool relevance pass. + // Some SessionStart payloads have no task signal. Fail closed for situational + // findings until the adapter supplies ids selected from actual task text. // Universal human/pinned rules are delivered separately by `standingRules`. if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read diff --git a/integrations/windsurf/hooks/lib/adapter.mjs b/integrations/windsurf/hooks/lib/adapter.mjs index 0e079eca7..78a5cd60d 100644 --- a/integrations/windsurf/hooks/lib/adapter.mjs +++ b/integrations/windsurf/hooks/lib/adapter.mjs @@ -52,6 +52,7 @@ import { forSharedCommand, forTouch, noteActClasses, + relevantFindingIdsForContext, sessionIndex, standingRules, } from './inject.mjs'; @@ -178,6 +179,24 @@ export function normalizeClientPayload(clientName, event, raw) { return raw; } +/** Extract only user/task text, never cwd, ids, or other metadata. */ +export function sessionTaskContext(raw = {}) { + return [ + raw.prompt, + raw.user_prompt, + raw.userPrompt, + raw.initial_prompt, + raw.initialPrompt, + raw.task_prompt, + raw.taskPrompt, + raw.task_description, + raw.taskDescription, + raw.task?.prompt, + raw.task?.text, + raw.message?.content, + ].filter((value) => typeof value === 'string' && value.trim()).join('\n'); +} + function emit(object) { process.stdout.write(JSON.stringify(object)); } @@ -439,7 +458,14 @@ export async function run(clientName, event) { const episode = episodeMeta({ client: clientName, raw }); const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); - const index = sessionIndex(dir, graph, { episode }); + const relevantFindingIds = relevantFindingIdsForContext( + graph, + sessionTaskContext(raw) + ); + const index = sessionIndex(dir, graph, { + episode, + relevantFindingIds, + }); if (index) parts.push(index); } catch { // Retrieval is optional context; the policy must still reach the model. diff --git a/integrations/windsurf/hooks/lib/inject.mjs b/integrations/windsurf/hooks/lib/inject.mjs index 1ec5b4ce4..b00494a01 100644 --- a/integrations/windsurf/hooks/lib/inject.mjs +++ b/integrations/windsurf/hooks/lib/inject.mjs @@ -772,6 +772,57 @@ export function forSharedCommand( } +const RELEVANCE_STOP_WORDS = new Set([ + 'about', 'after', 'again', 'before', 'could', 'from', 'have', 'into', + 'just', 'need', 'that', 'their', 'there', 'these', 'this', 'those', + 'using', 'want', 'when', 'where', 'which', 'with', 'would', 'your', +]); + +function relevanceTerms(value) { + return [...new Set(String(value || '').toLowerCase() + .split(/[^a-z0-9_.:/-]+/) + .map((term) => term.replace(/^[_.:/-]+|[_.:/-]+$/g, '')) + .filter((term) => term.length >= 3 && !RELEVANCE_STOP_WORDS.has(term)))]; +} + +/** + * Selects finding keys only when a lifecycle payload contains actual task text. + * Session-start envelopes that carry only cwd/session metadata deliberately + * return no ids; their first file/command event uses the contextual injectors. + */ +export function relevantFindingIdsForContext(graph, context, { limit = 8 } = {}) { + const query = String(context || '').trim(); + const queryTerms = relevanceTerms(query); + if (queryTerms.length < 2) return []; + + return [...graph.nodes.values()] + .filter((finding) => + finding.kind === 'finding' + && finding.retired !== true + && typeof finding.key === 'string' + && typeof finding.claim === 'string') + .map((finding) => { + const content = [ + finding.key, + finding.claim, + finding.trigger, + ...(finding.anchors || []), + ].filter(Boolean).join(' '); + const terms = new Set(relevanceTerms(content)); + const matches = queryTerms.filter((term) => terms.has(term)).length; + return { finding, matches }; + }) + // Two independent terms keeps a generic word such as "test" or "file" + // from turning the bounded index into an unrelated project-wide dump. + .filter(({ matches }) => matches >= 2) + .sort((a, b) => + b.matches - a.matches + || (b.finding.confidence || 0.5) - (a.finding.confidence || 0.5) + || a.finding.key.localeCompare(b.finding.key)) + .slice(0, Math.max(0, limit)) + .map(({ finding }) => finding.key); +} + /** * The bounded SessionStart index: titles and ids only, never bodies. * @@ -782,8 +833,8 @@ export function forSharedCommand( export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); const relevant = new Set(relevantFindingIds); - // SessionStart has no task signal. Fail closed for situational findings until - // a client supplies ids selected from an actual prompt/tool relevance pass. + // Some SessionStart payloads have no task signal. Fail closed for situational + // findings until the adapter supplies ids selected from actual task text. // Universal human/pinned rules are delivered separately by `standingRules`. if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read diff --git a/plugin/hooks/lib/adapter.mjs b/plugin/hooks/lib/adapter.mjs index 0e079eca7..78a5cd60d 100644 --- a/plugin/hooks/lib/adapter.mjs +++ b/plugin/hooks/lib/adapter.mjs @@ -52,6 +52,7 @@ import { forSharedCommand, forTouch, noteActClasses, + relevantFindingIdsForContext, sessionIndex, standingRules, } from './inject.mjs'; @@ -178,6 +179,24 @@ export function normalizeClientPayload(clientName, event, raw) { return raw; } +/** Extract only user/task text, never cwd, ids, or other metadata. */ +export function sessionTaskContext(raw = {}) { + return [ + raw.prompt, + raw.user_prompt, + raw.userPrompt, + raw.initial_prompt, + raw.initialPrompt, + raw.task_prompt, + raw.taskPrompt, + raw.task_description, + raw.taskDescription, + raw.task?.prompt, + raw.task?.text, + raw.message?.content, + ].filter((value) => typeof value === 'string' && value.trim()).join('\n'); +} + function emit(object) { process.stdout.write(JSON.stringify(object)); } @@ -439,7 +458,14 @@ export async function run(clientName, event) { const episode = episodeMeta({ client: clientName, raw }); const rules = standingRules(dir, graph, { episode }); if (rules) parts.push(rules); - const index = sessionIndex(dir, graph, { episode }); + const relevantFindingIds = relevantFindingIdsForContext( + graph, + sessionTaskContext(raw) + ); + const index = sessionIndex(dir, graph, { + episode, + relevantFindingIds, + }); if (index) parts.push(index); } catch { // Retrieval is optional context; the policy must still reach the model. diff --git a/plugin/hooks/lib/inject.mjs b/plugin/hooks/lib/inject.mjs index 1ec5b4ce4..b00494a01 100644 --- a/plugin/hooks/lib/inject.mjs +++ b/plugin/hooks/lib/inject.mjs @@ -772,6 +772,57 @@ export function forSharedCommand( } +const RELEVANCE_STOP_WORDS = new Set([ + 'about', 'after', 'again', 'before', 'could', 'from', 'have', 'into', + 'just', 'need', 'that', 'their', 'there', 'these', 'this', 'those', + 'using', 'want', 'when', 'where', 'which', 'with', 'would', 'your', +]); + +function relevanceTerms(value) { + return [...new Set(String(value || '').toLowerCase() + .split(/[^a-z0-9_.:/-]+/) + .map((term) => term.replace(/^[_.:/-]+|[_.:/-]+$/g, '')) + .filter((term) => term.length >= 3 && !RELEVANCE_STOP_WORDS.has(term)))]; +} + +/** + * Selects finding keys only when a lifecycle payload contains actual task text. + * Session-start envelopes that carry only cwd/session metadata deliberately + * return no ids; their first file/command event uses the contextual injectors. + */ +export function relevantFindingIdsForContext(graph, context, { limit = 8 } = {}) { + const query = String(context || '').trim(); + const queryTerms = relevanceTerms(query); + if (queryTerms.length < 2) return []; + + return [...graph.nodes.values()] + .filter((finding) => + finding.kind === 'finding' + && finding.retired !== true + && typeof finding.key === 'string' + && typeof finding.claim === 'string') + .map((finding) => { + const content = [ + finding.key, + finding.claim, + finding.trigger, + ...(finding.anchors || []), + ].filter(Boolean).join(' '); + const terms = new Set(relevanceTerms(content)); + const matches = queryTerms.filter((term) => terms.has(term)).length; + return { finding, matches }; + }) + // Two independent terms keeps a generic word such as "test" or "file" + // from turning the bounded index into an unrelated project-wide dump. + .filter(({ matches }) => matches >= 2) + .sort((a, b) => + b.matches - a.matches + || (b.finding.confidence || 0.5) - (a.finding.confidence || 0.5) + || a.finding.key.localeCompare(b.finding.key)) + .slice(0, Math.max(0, limit)) + .map(({ finding }) => finding.key); +} + /** * The bounded SessionStart index: titles and ids only, never bodies. * @@ -782,8 +833,8 @@ export function forSharedCommand( export function sessionIndex(dir, graph, { episode = {}, relevantFindingIds = [] } = {}) { const budget = indexBudget(dir); const relevant = new Set(relevantFindingIds); - // SessionStart has no task signal. Fail closed for situational findings until - // a client supplies ids selected from an actual prompt/tool relevance pass. + // Some SessionStart payloads have no task signal. Fail closed for situational + // findings until the adapter supplies ids selected from actual task text. // Universal human/pinned rules are delivered separately by `standingRules`. if (!relevant.size) return null; // RETIRED findings must not appear. They are excluded from every other read diff --git a/plugin/hooks/session-start.mjs b/plugin/hooks/session-start.mjs index eb51b3950..f795c386f 100644 --- a/plugin/hooks/session-start.mjs +++ b/plugin/hooks/session-start.mjs @@ -19,9 +19,13 @@ */ import { mode, MODE_OFF, readPayload, loadState } from './lib/policy.mjs'; -import { policyText } from './lib/adapter.mjs'; +import { policyText, sessionTaskContext } from './lib/adapter.mjs'; import { restorationPlan } from './lib/restore.mjs'; -import { sessionIndex, standingRules } from './lib/inject.mjs'; +import { + relevantFindingIdsForContext, + sessionIndex, + standingRules, +} from './lib/inject.mjs'; import { wikiDir, load, projectRootFor } from './lib/wiki.mjs'; import { episodeMeta, featuresForArm } from './lib/experiment.mjs'; import { join } from 'node:path'; @@ -103,8 +107,13 @@ try { const graph = load(dir); const rules = standingRules(dir, graph); if (rules) parts.push(rules); + const relevantFindingIds = relevantFindingIdsForContext( + graph, + sessionTaskContext(payload) + ); const index = sessionIndex(dir, graph, { episode: episodeMeta({ client: 'claude-code', raw: payload }), + relevantFindingIds, }); if (index) parts.push(index); } diff --git a/tests/hooks/injection.test.mjs b/tests/hooks/injection.test.mjs index a6db68350..a5db73e18 100644 --- a/tests/hooks/injection.test.mjs +++ b/tests/hooks/injection.test.mjs @@ -11,7 +11,13 @@ import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'; import { join } from 'node:path'; import { tmpdir } from 'node:os'; import { validate, buildDigest, harvestEnabled } from '../../hooks-core/harvest.mjs'; -import { forTouch, sessionIndex, refusalPayload, linkCoOccurrence } from '../../hooks-core/inject.mjs'; +import { + forTouch, + relevantFindingIdsForContext, + sessionIndex, + refusalPayload, + linkCoOccurrence, +} from '../../hooks-core/inject.mjs'; import { inHoldout, record, readMetrics, report, indexBudget } from '../../hooks-core/metrics.mjs'; import { load, putNode, putEdge, nodeId } from '../../hooks-core/wiki.mjs'; import { indexFile } from '../../hooks-core/staleness.mjs'; @@ -160,6 +166,24 @@ describe('P4 injection', () => { expect(delivery.deliveredTokens).toBeGreaterThan(0); }); + test('task text selects only situational findings with two-term evidence', () => { + const path = write('commands.ts', 'export const runner = "npm test";'); + indexFile(dir, path); + seedFinding(path, 'Run npm test instead of npx jest for this repository'); + seedFinding(path, 'The deployment region is selected by the release job'); + + const graph = load(dir); + const selected = relevantFindingIdsForContext( + graph, + 'The npx jest command failed; fix the repository test runner.' + ); + expect(selected).toHaveLength(1); + expect( + [...graph.nodes.values()].find((node) => node.key === selected[0]).claim + ).toContain('npm test'); + expect(relevantFindingIdsForContext(graph, 'test')).toEqual([]); + }); + test('the session index labels invalidated content claims before delivery', () => { const path = write('stale.ts', 'export const mode = "old";'); indexFile(dir, path); diff --git a/tests/hooks/standing-rules.test.mjs b/tests/hooks/standing-rules.test.mjs index 595496d4e..51aaf77e6 100644 --- a/tests/hooks/standing-rules.test.mjs +++ b/tests/hooks/standing-rules.test.mjs @@ -12,6 +12,7 @@ */ import { describe, it, expect, beforeEach, afterEach } from '@jest/globals'; import { sessionIndex, standingRules } from '../../hooks-core/inject.mjs'; +import { readMetrics } from '../../hooks-core/metrics.mjs'; import { load, putNode, wikiDir } from '../../hooks-core/wiki.mjs'; import { spawnSync } from 'child_process'; import { mkdtempSync, rmSync, readFileSync, mkdirSync } from 'fs'; @@ -176,6 +177,40 @@ describe('through the real SessionStart hook', () => { expect(ctx).not.toContain('# Project wiki'); expect(ctx).not.toContain('npx jest'); + // Both the custom Claude entry and the generated shared adapter now pass + // prompt-selected finding ids through the same fail-closed contract. + for (const entry of [ + join(process.cwd(), 'plugin', 'hooks', 'session-start.mjs'), + join(process.cwd(), 'integrations', 'codex', 'hooks', 'session-start.mjs'), + ]) { + const relevantRun = spawnSync(process.execPath, [entry], { + input: JSON.stringify({ + cwd: project, + userPrompt: 'The npx jest command failed; fix the test runner.', + }), + encoding: 'utf8', + timeout: 30_000, + env: { + ...process.env, + TOKEN_OPTIMIZER_WIKI_DIR: graphDir, + TOKEN_OPTIMIZER_SHARED_DIR: graphDir, + CLAUDE_PROJECT_DIR: project, + }, + }); + expect(relevantRun.status).toBe(0); + expect(relevantRun.stdout).toContain('# Project wiki'); + expect(relevantRun.stdout).toContain('npx jest'); + } + const adapterDeliveries = readMetrics(graphDir).filter( + (event) => + event.kind === 'inject' && + event.surface === 'session-start' && + event.findingIds?.includes('c1') + ); + expect(new Set(adapterDeliveries.map((event) => event.client))).toEqual( + new Set(['claude-code', 'codex']) + ); + // A prompt/tool relevance pass can explicitly select the situational item. const relevant = sessionIndex(graphDir, load(graphDir), { relevantFindingIds: ['c1'] }); expect(relevant).toContain('# Project wiki');