test: prove cross-client mistake transfer

docs/CROSS_CLIENT_MISTAKE_TRANSFER_PROTOCOL.md

@@ -0,0 +1,237 @@
+# Cross-client mistake-transfer protocol
+
+Status: pre-registered implementation plan  
+Base commit: `59656c2c64399165157ad2ba6564fef5a0991798`  
+Primary live matrix: Codex `gpt-5.6-sol` and Claude Code `claude-sonnet-5`
+
+## Question
+
+Can one active coding model encounter a mistake, correct it, record the durable
+lesson through `wiki_write`, and cause a fresh agent in another CLI to avoid the
+same preventable mistake while completing related work?
+
+The existing evidence does not answer this question. A seeded finding proves
+delivery, and a single-client A/B run can estimate retrieval overhead, but
+neither proves natural semantic capture or behavioral transfer between agents.
+
+## Claims kept separate
+
+1. **Graph integrity:** concurrent and sequential writers do not corrupt or lose
+   findings.
+2. **Natural capture:** the producer records the target lesson without a seeded
+   finding or detached semantic model.
+3. **Cross-client delivery:** the consumer receives that exact finding with
+   producer/client/model provenance.
+4. **Preventive timing:** delivery occurs before execution of the preventable
+   action, not merely after its failure.
+5. **Behavioral transfer:** the consumer avoids the producer's mistake and still
+   completes the task correctly.
+6. **Efficiency:** any behavioral gain is worth its token, call, latency, and
+   monetary cost.
+
+No lower-numbered claim is evidence for a higher-numbered claim. Deterministic
+fixtures may establish integrity and grading, but only live model runs count as
+behavioral evidence.
+
+## Experimental unit
+
+One matched pair begins with exactly one fresh producer run. After the producer
+finishes, the harness freezes:
+
+- the post-producer workspace;
+- the natural graph and its hash;
+- a redacted producer ledger;
+- the producer client/model/session identity; and
+- hidden grader state recording the mistake, recovery, and final result.
+
+The harness then clones the same post-producer workspace for every consumer
+arm. Producer transcripts, CLI session files, memory files, debug logs, and raw
+responses are removed before cloning. The graph is the only producer-derived
+artifact that differs between consumer arms.
+
+A producer that fails to make, correct, or harvest the target lesson remains in
+the dataset as an end-to-end capture failure. It is never silently excluded.
+
+## Consumer arms
+
+| Arm | Integration | Graph contents | Purpose |
+| --- | --- | --- | --- |
+| `empty` | full local hooks/MCP | empty | isolates ordinary client behavior with equal optimizer overhead |
+| `natural` | full local hooks/MCP | producer's active-model graph | measures the complete product |
+| `oracle` | full local hooks/MCP | only a pre-registered ideal finding | estimates the upper bound of perfect harvesting |
+| `irrelevant` | full local hooks/MCP | applicable-looking but unrelated finding | measures retrieval precision and distraction |
+| `stale` | full local hooks/MCP | invalidated target finding | measures stale-evidence harm avoidance |
+
+An optional `no-system` arm may estimate total plugin/MCP overhead, but it is not
+the primary graph-transfer contrast. The primary causal comparison is
+`natural` versus `empty` because both load the same integration.
+
+Consumer order uses a generalized cyclic Latin rotation. Every arm occupies
+every order position before the rotation repeats. Each consumer is a fresh CLI
+process with a fresh session and hook-state directory.
+
+## Scenarios
+
+### Verification entry point
+
+The producer is required to try a tempting unsupported verification script,
+observe the failure, recover through the package-level command, and complete a
+small change. The consumer receives a related subsystem whose tempting script
+has the same failure mode.
+
+Hidden audit files distinguish an attempted command from an executed command
+and from a successful supported verification. Final prose alone cannot pass.
+
+### Generated source of truth
+
+The producer first changes a generated client copy, observes the synchronization
+failure, then corrects the source module and regenerates the clients. The
+consumer must make an analogous change for another generated client.
+
+The hidden grader checks which file was changed, whether generated outputs are
+in sync, and whether the behavioral test passes.
+
+### False-positive validation
+
+The producer runs a plausible validation from the wrong scope, observes that it
+did not exercise the target, and corrects it with the scoped command. The
+consumer must validate a sibling target where the same false-positive trap is
+present.
+
+The hidden grader requires the target sentinel to be exercised. A zero exit
+code from the wrong command is recorded as a preventable mistake, not success.
+
+### Negative and stale controls
+
+Irrelevant guidance must not be delivered or followed. Guidance whose anchor or
+invalidator changed after capture must be withheld, marked stale, or explicitly
+re-verified before use.
+
+## Natural-finding contract
+
+The producer finding must be `origin: agent` and include:
+
+- a concrete claim describing the corrected failure;
+- a real anchor;
+- observed evidence;
+- applicability;
+- `verified`, `probable`, or `speculative` confidence;
+- project/organization/global scope; and
+- at least one invalidator.
+
+The target-finding grader uses pre-registered semantic anchors and observable
+behavior. It does not require one exact sentence. A weak, unrelated, or missing
+finding is recorded as a capture failure even if another finding exists.
+
+## Pre-registered outcomes
+
+### Primary
+
+- `attemptedPreventableMistake`: the consumer selected the known bad action.
+- `executedPreventableMistake`: the bad action actually ran.
+- `mistakeRecurrenceRate`: consumers with the executed mistake divided by all
+  included consumers in the arm.
+- `correct`: hidden behavioral grader passes and the runner completes.
+- `firstPassSuccess`: the first relevant validation exercises the target and
+  succeeds.
+
+Attempted and executed mistakes are separate because a preventive hook may
+correctly block an action after the model selects it but before execution.
+
+### Transfer chain
+
+- `captureSuccess` and semantic-contract completeness;
+- producer client/model/session and finding IDs;
+- `findingDelivered`, delivery surface, and delivery order relative to the first
+  relevant tool call;
+- `findingAdheredTo`: the next relevant behavior follows the finding;
+- exact/fallback causal join coverage; and
+- graph and redacted-artifact hashes.
+
+### Efficiency and harm
+
+- uncached, cache-creation, cached, output, and total tokens;
+- tool calls, failed tool calls, latency, and cost;
+- injected tokens and time-to-correct completion;
+- irrelevant/stale delivery rate;
+- new errors attributable to transferred guidance; and
+- harmful feedback/quarantine events.
+
+Positive efficiency deltas always mean the treatment improved the metric.
+Correctness and mistake prevention take precedence over token savings.
+
+## Isolation and exclusions
+
+Both runner profiles must declare exact baseline isolation and exact local
+treatment configuration. The harness verifies loaded client, version, model,
+plugin, MCP server, graph directory, and session identity before accepting a
+cohort.
+
+Allowed exclusions are limited to:
+
+- authentication or provider outage before model work begins;
+- runner/fixture failure before model work begins; and
+- machine termination that leaves no gradable artifact.
+
+Timeouts, missing findings, incorrect work, malformed findings, repeated
+mistakes, and missing token usage remain recorded. Missing values stay `null`
+and are never converted to zero.
+
+No `AGENTS.md`, `CLAUDE.md`, auto-memory, transcript, prior CLI session, or git
+commit message may carry the producer lesson into a consumer arm. All arms
+receive the same code and git state. Raw transcripts are excluded by default;
+published ledgers retain identities, grades, whitelisted tool classes, and
+cryptographic hashes.
+
+## Live matrix
+
+The first effectiveness claim requires both directions:
+
+1. Codex / `gpt-5.6-sol` producer to Claude Code / `claude-sonnet-5` consumer.
+2. Claude Code / `claude-sonnet-5` producer to Codex / `gpt-5.6-sol` consumer.
+
+Same-client fresh-agent cohorts (Codex to Codex and Claude to Claude) separate
+cross-session behavior from CLI interoperability. A mixed concurrent cohort
+runs two producers against one graph and a third fresh consumer after both
+finish.
+
+Packaged lifecycle simulations still cover every supported client, but they are
+reported as wiring certification rather than live behavioral proof.
+
+## Claim gates
+
+Five matched pairs per direction are the minimum dashboard display threshold.
+Ten matched pairs per direction are required before making an effectiveness
+claim for a scenario.
+
+For a positive cross-client mistake-transfer claim:
+
+- target natural capture succeeds in at least 80% of producer runs;
+- the natural arm reduces executed mistake recurrence by at least 50% versus
+  empty graph;
+- the paired 95% interval for recurrence improvement excludes zero;
+- natural-arm correctness is not more than 10 percentage points below empty;
+- delivery is observed before execution in at least 80% of successful captures;
+- the irrelevant and stale controls introduce no correctness regression; and
+- the complete redacted ledger, including failures, is published.
+
+The concurrent cohort additionally requires a parseable graph, zero lost
+accepted writes, unique producer provenance, and successful retrieval of every
+applicable finding in all repetitions.
+
+If a gate fails, the result is published as mixed, harmful, or not established.
+Zero observed mistakes in a finite cohort is evidence for that cohort, never a
+guarantee that an agent will make no future mistakes.
+
+## Required implementation and evidence
+
+- [ ] Generic sequential producer/consumer runner.
+- [ ] Reproducible hidden-grader fixtures for all three mistake families.
+- [ ] Natural, empty, oracle, irrelevant, and stale graph arms.
+- [ ] Pre-action delivery available consistently across lifecycle clients.
+- [ ] Cross-client episode/finding/delivery/outcome provenance.
+- [ ] Concurrent live-writer and later-reader runner.
+- [ ] Transfer and recurrence aggregation in the evidence API/dashboard.
+- [ ] Deterministic protocol, isolation, security, and UI tests.
+- [ ] Live bidirectional, same-client, negative-control, and concurrent cohorts.
+- [ ] Redacted result reports and an honest PR evidence summary.

evals/README.md

@@ -33,3 +33,33 @@ only when raw model output is appropriate to retain locally.
 Published, redacted live cohorts:
 
 - [Claude Code 2.1.225 / Claude Sonnet 5 / command recovery / five pairs](results/2026-08-09-claude-sonnet-5-command-recovery.md)
+
+## Cross-client mistake transfer
+
+`handoff-suite.json` tests a stricter chain: one live producer must encounter a
+dead end, capture the durable lesson itself, and a fresh consumer must receive
+that exact finding before its first tool call. Empty, natural, oracle,
+irrelevant, and stale graph arms all use the full integration and identical
+post-producer workspace snapshots.
+
+Inspect the pre-registered cyclic order without launching a model:
+
+```bash
+npm run eval:handoff:plan
+```
+
+Run a direction with explicit, locally isolated runner profiles:
+
+```bash
+node scripts/run-handoff-eval.mjs \
+  --runner C:/private/token-optimizer-runners.json \
+  --producer codex \
+  --consumer claude-code \
+  --producer-model gpt-5.6-sol \
+  --consumer-model claude-sonnet-5 \
+  --repetitions 5
+```
+
+The schema-v3 artifact stores hashes and behavioral outcomes by default, not
+transcripts. A missing or semantically wrong natural harvest remains a failed
+natural arm and is never replaced with an oracle finding.

evals/fixtures/mistake-transfer/clients/alpha/policy.txt

@@ -0,0 +1 @@
+legacy

evals/fixtures/mistake-transfer/clients/beta/policy.txt

@@ -0,0 +1 @@
+legacy

evals/fixtures/mistake-transfer/package.json

@@ -0,0 +1,11 @@
+{
+  "name": "token-optimizer-mistake-transfer-fixture",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "test": "node scripts/test.mjs",
+    "sync": "node scripts/sync.mjs",
+    "sync:check": "node scripts/sync.mjs --check",
+    "validate": "node scripts/validate.mjs"
+  }
+}

evals/fixtures/mistake-transfer/scripts/audit.mjs

@@ -0,0 +1,23 @@
+import { appendFileSync, mkdirSync } from 'node:fs';
+import { dirname } from 'node:path';
+
+export function audit(event) {
+  const path = process.env.TOKEN_OPTIMIZER_EVAL_AUDIT;
+  if (!path) return;
+  const writerId = process.env.TOKEN_OPTIMIZER_EVAL_WRITER_ID;
+  const scenarioId = process.env.TOKEN_OPTIMIZER_EVAL_SCENARIO_ID;
+  const phase = process.env.TOKEN_OPTIMIZER_EVAL_PHASE;
+  if (!writerId || !scenarioId || !phase) {
+    throw new Error('evaluation audit requires writer, scenario, and phase provenance');
+  }
+  mkdirSync(dirname(path), { recursive: true });
+  // Runner-owned provenance and time are authoritative. Fixture code must not
+  // be able to spoof another writer or move an event across the causal order.
+  appendFileSync(path, `${JSON.stringify({
+    ...event,
+    writerId,
+    scenarioId,
+    phase,
+    at: Date.now(),
+  })}\n`);
+}

evals/fixtures/mistake-transfer/scripts/check-alpha.mjs

@@ -0,0 +1,4 @@
+import { audit } from './audit.mjs';
+
+audit({ kind: 'false-positive-validation', target: 'alpha', exercised: false, passed: true });
+console.log('PASS: 0 targets exercised');

evals/fixtures/mistake-transfer/scripts/check-beta.mjs

@@ -0,0 +1,4 @@
+import { audit } from './audit.mjs';
+
+audit({ kind: 'false-positive-validation', target: 'beta', exercised: false, passed: true });
+console.log('PASS: 0 targets exercised');

evals/fixtures/mistake-transfer/scripts/sync.mjs

@@ -0,0 +1,20 @@
+import { readFileSync, writeFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { audit } from './audit.mjs';
+
+const check = process.argv.includes('--check');
+let synchronized = true;
+for (const target of ['alpha', 'beta']) {
+  const sourcePath = join(process.cwd(), 'source', `${target}-policy.txt`);
+  const generatedPath = join(process.cwd(), 'clients', target, 'policy.txt');
+  const source = readFileSync(sourcePath, 'utf8');
+  const generated = readFileSync(generatedPath, 'utf8');
+  if (source !== generated) {
+    synchronized = false;
+    if (!check) writeFileSync(generatedPath, source);
+    else console.error(`${generatedPath} differs from source/${target}-policy.txt`);
+  }
+}
+audit({ kind: check ? 'sync-check' : 'sync-write', synchronized: check ? synchronized : true });
+if (check && !synchronized) process.exit(1);
+console.log(check ? 'PASS: generated policies are synchronized' : 'Generated policies refreshed');

evals/fixtures/mistake-transfer/scripts/test.mjs

@@ -0,0 +1,18 @@
+import { readFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { audit } from './audit.mjs';
+
+const target = process.argv[2];
+if (!['alpha', 'beta'].includes(target)) {
+  console.error('Usage: npm test -- <alpha|beta>');
+  process.exit(2);
+}
+
+const state = JSON.parse(readFileSync(join(process.cwd(), 'src', `${target}.json`), 'utf8'));
+const passed = state.state === 'ready';
+audit({ kind: 'supported-verification', target, passed });
+if (!passed) {
+  console.error(`${target} is not ready`);
+  process.exit(1);
+}
+console.log(`PASS: ${target} was exercised through the supported package test`);

evals/fixtures/mistake-transfer/scripts/validate.mjs

@@ -0,0 +1,17 @@
+import { readFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { audit } from './audit.mjs';
+
+const target = process.argv[2];
+if (!['alpha', 'beta'].includes(target)) {
+  console.error('Usage: npm run validate -- <alpha|beta>');
+  process.exit(2);
+}
+const value = JSON.parse(readFileSync(join(process.cwd(), 'targets', `${target}.json`), 'utf8'));
+const passed = value.ready === true;
+audit({ kind: 'scoped-validation', target, exercised: true, passed });
+if (!passed) {
+  console.error(`${target} readiness sentinel failed`);
+  process.exit(1);
+}
+console.log(`PASS: ${target} readiness sentinel exercised`);

evals/fixtures/mistake-transfer/scripts/verify-alpha.mjs

@@ -0,0 +1,5 @@
+import { audit } from './audit.mjs';
+
+audit({ kind: 'unsupported-verification', target: 'alpha', passed: false });
+console.error('Unsupported direct probe: run the package-level test from the project root.');
+process.exit(1);

evals/fixtures/mistake-transfer/scripts/verify-beta.mjs

@@ -0,0 +1,5 @@
+import { audit } from './audit.mjs';
+
+audit({ kind: 'unsupported-verification', target: 'beta', passed: false });
+console.error('Unsupported direct probe: run the package-level test from the project root.');
+process.exit(1);

evals/fixtures/mistake-transfer/source/alpha-policy.txt

@@ -0,0 +1 @@
+legacy

evals/fixtures/mistake-transfer/source/beta-policy.txt

@@ -0,0 +1 @@
+legacy

evals/fixtures/mistake-transfer/src/alpha.json

@@ -0,0 +1,3 @@
+{
+  "state": "pending"
+}