diff --git a/.github/workflows/quality-gates.yml b/.github/workflows/quality-gates.yml index 7ea37ae9a..4913f35a2 100644 --- a/.github/workflows/quality-gates.yml +++ b/.github/workflows/quality-gates.yml @@ -68,6 +68,9 @@ jobs: - name: Verify Universal Cognitive Runtime conformance and evidence run: npm run verify:ucr + - name: Verify full UCR study and metric coverage + run: npm run verify:ucr:study-design + # BROWSER GATES. These verify the wiki dashboard actually works -- they # caught a real defect the moment they were first run: an open detail panel # overlaid the toolbar and list, so those controls could not be clicked at diff --git a/README.md b/README.md index 309c2a18e..7b3aff19c 100644 --- a/README.md +++ b/README.md @@ -1413,13 +1413,25 @@ consumers normally receive zero MCP tools: host pre-action delivery adds only the selected capsule through the client lifecycle channel. Other enabled MCP servers add their own schemas independently. -The current two-direction Codex/Claude executable smoke records 21.06% lower -combined predecessor-to-successor token traffic, 18.36% lower combined latency, -one known-mistake recurrence in blinded control versus zero in runtime, and one -native pre-execution denial. These are signed smoke observations, not powered -effectiveness or universal-superiority claims; the evidence index deliberately -keeps the release verdict `insufficient` until its confidence, competitive, and -production gates pass. +The current hardened cross-CLI smoke does not qualify. In the final +Codex-to-Claude adversarial pair, both successors were correct and the runtime +capsule was delivered with exact provider model attestations and zero consumer +MCP tools, but runtime used 5.383% more tokens and 41.758% more latency. The +reciprocal runtime arm was blocked by Claude provider quota, and Antigravity +1.1.11 required account authentication before a Gemini model could run. These +signed outcomes keep the release verdict `insufficient`; they are evidence of +working provenance and fail-closed behavior, not powered effectiveness or +universal superiority. + +The replacement-grade protocol is intentionally larger than that smoke: +54,054 all-family/all-arm trial envelopes, 113,022 provider +calls, three independent model families, direction-level non-inferiority, and +1,056 hard-negative opportunities per direction and arm so the Bonferroni-adjusted +95% family-wise false-delivery upper bound—not only the point estimate—must +remain below 1%. Run +`npm run verify:ucr:study-design` to validate the frozen metric coverage. The +full protocol and CLI driver contract are in +[`evals/ucr/FULL_STUDY_CONTRACT.md`](evals/ucr/FULL_STUDY_CONTRACT.md). #### Performance Controls diff --git a/docs/UNIVERSAL_COGNITIVE_RUNTIME_PROGRAM.md b/docs/UNIVERSAL_COGNITIVE_RUNTIME_PROGRAM.md index fceeb8f99..986d3232d 100644 --- a/docs/UNIVERSAL_COGNITIVE_RUNTIME_PROGRAM.md +++ b/docs/UNIVERSAL_COGNITIVE_RUNTIME_PROGRAM.md @@ -966,32 +966,40 @@ traffic studies pass. The evidence contract has six non-interchangeable classes: transport, conformance, executable smoke, effectiveness, superiority, and production. -`evals/ucr/results/evidence-index-v2.json` verifies 12/12 current artifacts and +`evals/ucr/results/evidence-index-v2.json` verifies 13/13 current artifacts and contains signed, hash-chained, redacted ledgers. Conformance and executable smoke are present; effectiveness, superiority, and production remain missing. +The executable promotion protocol is defined in +[`evals/ucr/FULL_STUDY_CONTRACT.md`](../evals/ucr/FULL_STUDY_CONTRACT.md). It +separates effectiveness, competitive superiority, and production readiness; +maps every release metric in CI; grades fresh hidden filesystem variants outside +the provider process; covers all families, arms, directions, sessions, projects, +and concurrency modes; and powers the per-direction hard-negative confidence +bound. This design coverage is not itself live evidence. + ## Workstream evidence matrix -| # | Implemented in PR #303 | Evidence now | Program exit still open | -| --: | ---------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ | -| 1 | Signed run identity, versioned events, content-addressed artifacts, migrations, canonical replay | 29/29 runtime checks plus byte-equivalent independent Python replay | Production redacted replay corpus | -| 2 | Typed bitemporal/causal graph and streaming validated projection | 1,000,000 events, 44,475 events/s, zero duplicates/orphans/diagnostics | Full legacy-wiki migration audit | -| 3 | In-turn model-authored semantic delta, authenticate-before-activate receipts, canonical dedupe/supersession, hard negatives | Zero detached capture calls in both live directions; malformed, oversized, forged, or inapplicable deltas are rejected | Powered semantic-quality corpus | -| 4 | Scoped guard DSL, signed active index, trigger matching, audit, tamper fail-open, native adapter wiring | Ten native adapters pass conformance; one live Codex pre-tool action denied before execution | Powered live prevention study | -| 5 | Exact causal joins, ablations, confounder refusal, quarantine latency | Confounded credit rejected; counterfactual and quarantine tests pass | Matched live causal cohorts | -| 6 | L0-L4 Context VM, single compact correction, phase ledger, persistent working sets, deltas, zero-work starts | 64/91-token live capsules; 21.06% lower combined full pipeline traffic across two smoke pairs | Paired live empty-session p50/p95 and powered interval | -| 7 | Complete signed checkpoints, atomic storage, deltas, takeover/staleness study | Stale workspace rejected before action; multi-family fixture study | Powered live takeover non-inferiority | -| 8 | SQLite/WAL tasks, transactions, leases, recovery, conflict and partition reconciliation | 100 physical workers, zero lost writes, 100% duplicate suppression | Multi-host production contention | -| 9 | Immutable consolidation proposals, contradictions, decay, bounded growth | 100 sessions, zero source mutations, 88/88 delayed reuse retained | Powered semantic retention quality | -| 10 | Negotiation, authenticated principals, taint/redaction, revocation, red-team transfer suite | Signed transfer accepted; injection/revoked transfer denied | External security assessment | -| 11 | BM25, vector, temporal, causal, structural, procedure, checkpoint, and global kernels with calibrated routing | Kernel, abstention, compatibility, and calibration tests pass | Frozen live recall/cost/latency comparison | -| 12 | Two-operation continuity MCP surface plus optional extended cognitive profile with real `tools/list` native-token audit | 480 tokens versus 4,815 core and 30,125 full; 98.41% reduction versus full; live successors expose zero MCP tools | Paired provider prompt-overhead distribution | -| 13 | One adapter SDK, native UCR guards, and honest capability tiers for 16 clients | 16 child processes certified; ten generated native guard adapters; live Codex/Claude bidirectional smoke | Paid native CLI execution for unavailable clients | -| 14 | Frozen 11-family, seven-arm benchmark, hidden variants, signed graders, preregistration and power analysis | 11 natural fixtures; required sample size 357 per arm | Full powered model execution | -| 15 | Ten executable reference implementations, fairness and Pareto gates | 1,000 reference cases; all labelled non-product baselines | Fair strongest-product reproductions | -| 16 | Linked 100-task/700-arm curriculum, immutable ledger, learning and explicit ablations | 0.88 recurrence and 0.65 reconstruction reductions in deterministic policy fixtures | Paid live compounding execution | -| 17 | Tiered evidence contract, phase accounting, release gate, API, dashboard, blinded four-call cross-CLI harness | 12/12 artifacts valid; 2/2 primary directions correct; control recurrence 1 versus runtime 0; dashboard exposes efficiency and guard metrics | Powered effectiveness and third-family provider availability | -| 18 | SLO aggregation, six-class fault injection, staged rollout, scoped kills, safe mode, breaker, recovery, readiness gate | 6/6 local faults contained, zero event loss, signed ledger | Staged production traffic and production-tier SLO window | +| # | Implemented in PR #303 | Evidence now | Program exit still open | +| --: | ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------ | +| 1 | Signed run identity, versioned events, content-addressed artifacts, migrations, canonical replay | 29/29 runtime checks plus byte-equivalent independent Python replay | Production redacted replay corpus | +| 2 | Typed bitemporal/causal graph and streaming validated projection | 1,000,000 events, 44,475 events/s, zero duplicates/orphans/diagnostics | Full legacy-wiki migration audit | +| 3 | In-turn model-authored semantic delta, authenticate-before-activate receipts, canonical dedupe/supersession, hard negatives | Zero detached capture calls in both live directions; malformed, oversized, forged, or inapplicable deltas are rejected | Powered semantic-quality corpus | +| 4 | Scoped guard DSL, signed active index, trigger matching, audit, tamper fail-open, native adapter wiring | Ten native adapters pass conformance; one live Codex pre-tool action denied before execution | Powered live prevention study | +| 5 | Exact causal joins, ablations, confounder refusal, quarantine latency | Confounded credit rejected; counterfactual and quarantine tests pass | Matched live causal cohorts | +| 6 | L0-L4 Context VM, single compact correction, phase ledger, persistent working sets, deltas, zero-work starts | Matched live smoke: 5.66% lower combined full-pipeline traffic and 4.73% higher latency; no correctness-effect opportunity | Paired live empty-session p50/p95 and powered interval | +| 7 | Complete signed checkpoints, atomic storage, deltas, takeover/staleness study | Stale workspace rejected before action; multi-family fixture study | Powered live takeover non-inferiority | +| 8 | SQLite/WAL tasks, transactions, leases, recovery, conflict and partition reconciliation | 100 physical workers, zero lost writes, 100% duplicate suppression | Multi-host production contention | +| 9 | Immutable consolidation proposals, contradictions, decay, bounded growth | 100 sessions, zero source mutations, 88/88 delayed reuse retained | Powered semantic retention quality | +| 10 | Negotiation, authenticated principals, taint/redaction, revocation, red-team transfer suite | Signed transfer accepted; injection/revoked transfer denied | External security assessment | +| 11 | BM25, vector, temporal, causal, structural, procedure, checkpoint, and global kernels with calibrated routing | Kernel, abstention, compatibility, and calibration tests pass | Frozen live recall/cost/latency comparison | +| 12 | Two-operation continuity MCP surface plus optional extended cognitive profile with real `tools/list` native-token audit | 480 tokens versus 4,815 core and 30,125 full; 98.41% reduction versus full; live successors expose zero MCP tools | Paired provider prompt-overhead distribution | +| 13 | One adapter SDK, native UCR guards, and honest capability tiers for 16 clients | 16 child processes certified; ten generated native guard adapters; live Codex/Claude bidirectional smoke | Paid native CLI execution for unavailable clients | +| 14 | Frozen 11-family, seven-arm benchmark, hidden variants, parent-owned filesystem graders, preregistration and power analysis | Full 54,054-trial plan; 363 paired observations and 1,056 negative opportunities per direction/arm; family-wise CI maps every metric | Full 113,022-call provider execution | +| 15 | Ten executable reference implementations, fairness and Pareto gates | 1,000 reference cases; all labelled non-product baselines | Fair strongest-product reproductions | +| 16 | Linked 100-task/700-arm curriculum, immutable ledger, learning and explicit ablations | 0.88 recurrence and 0.65 reconstruction reductions in deterministic policy fixtures | Paid live compounding execution | +| 17 | Independent effectiveness/superiority/production verdicts, phase accounting, confidence gates, dashboard, and universal study driver | 13/13 current artifacts valid; powered design/metric coverage passes; dashboard exposes tier and directional safety gaps | Powered effectiveness and third-family provider availability | +| 18 | SLO aggregation, six-class fault injection, staged rollout, scoped kills, safe mode, breaker, recovery, readiness gate | 6/6 local faults contained, zero event loss, signed ledger | Staged production traffic and production-tier SLO window | ## PR #303 implementation task checklist diff --git a/evals/ucr/FULL_STUDY_CONTRACT.md b/evals/ucr/FULL_STUDY_CONTRACT.md new file mode 100644 index 000000000..e8c496d8f --- /dev/null +++ b/evals/ucr/FULL_STUDY_CONTRACT.md @@ -0,0 +1,189 @@ +# Full UCR effectiveness, superiority, and production contract + +The full program has three independent signed verdicts. Passing mechanism or +adapter tests cannot promote any of them. + +## 1. Effectiveness + +`npm run eval:ucr:full-study:powered-plan` freezes the complete study before any +provider call. The current design contains: + +- all 11 benchmark families and all seven context arms; +- same-client and bidirectional cross-client handoffs across Codex, Claude Code, + and a third independent model family; +- same-session, cross-session, same-project, cross-project, single-agent, and + concurrent-agent trials; +- 363 paired empty/runtime observations per direction, exceeding the + preregistered 357-pair correctness requirement; +- 1,056 opportunities per direction for each hard-negative arm. With zero + false deliveries, the Bonferroni-adjusted 95% family-wise Wilson upper bound + across 9 directions × 5 safety arms is below 1%; and +- a unique hidden variant, session, and mutable workspace for every arm. + +The plan has 54,054 trial envelopes and 113,022 provider calls. Ordinary envelopes require a matched +predecessor and successor; concurrent-coordination envelopes require one +producer plus two overlapping successors. The larger count is deliberate: the +previous handoff plan made only 2,856 calls, repeated one handoff task, and could +not establish family coverage or a sub-1% negative-delivery confidence bound. + +The parent process materializes each repository fixture and grades its filesystem +state after the model exits. Provider prose and claimed success are ignored. +Protected fixture changes fail the grade. Raw prompts, transcripts, and model +output are rejected anywhere in a signed evidence row, including nested fields. + +## Driver contract for every CLI + +Every direction driver receives one `ucr.study-trial-request/1` JSON object on +stdin and returns one `ucr.study-driver/1` object on stdout. The driver must: + +1. invoke the producer on a fresh copy of the fixture; +2. let that active model author the compact semantic delta during its work turn; +3. let the host authenticate and activate or withhold that delta according to + the randomized arm, with zero capture-only model calls; +4. run mandatory bounded pre-action retrieval; +5. invoke one fresh consumer, or two host-observed overlapping successors for + concurrent-coordination trials, with zero MCP tools and zero static MCP schema; +6. retain complete pre-tool action telemetry; and +7. return provider-native token, cache, latency, cost, CLI/model version, and + observed session/project/agent topology telemetry for every call, plus the + executed top-level `promptHash`, `permissionsHash`, and `budgets` bindings; + and +8. bind every invocation to a hashed `ucr.model-attestation/1` record from its + provider-native stream. The requested and effective models, request/session + id, transport, and reroute history must match the frozen trial exactly. + +The frozen transports are Codex app-server, Claude Code stream JSON, and Google +Antigravity stream JSON. A separate Gemini CLI API-key registry is available for +operators who intentionally preregister that transport. Codex app-server runs +with a disposable `CODEX_HOME`, no MCP servers or inherited plugins, exact-model +fallback disabled, and only path-confined evaluation tools. A missing provider +model id, request binding, token record, or model reroute fails the trial closed. + +Drivers may attest only the capture-through-use portion of the causal chain. +The parent grader alone appends behavior change, mistake prevention, and task +correctness after comparing the paired filesystem outcomes; model or driver +claims cannot populate those outcome stages. + +The dispatcher reads a direction-specific environment variable such as +`UCR_STUDY_DIRECTION_CLAUDE_CODE_TO_CODEX`. Every registered CLI uses the same +protocol; lifecycle capability and live certification remain separate facts. +Missing executables, quotas, versions, action audits, native usage data, or +independent grades fail the trial and remain in the append-only attempt ledger. + +Run a non-promotable qualification first: + +```text +npm run eval:ucr:full-study:qualification +``` + +The repository ships `scripts/ucr-live-study-driver.mjs`, an executable +provider-CLI implementation of the shared driver protocol. Provision the +operator-owned signing identity and hidden-variant secret outside the repository +before qualification: + +```text +npm run eval:ucr:identity:provision +``` + +The full-study runner also accepts comma-separated `--directions`, `--families`, +and `--arms` selectors for bounded qualification. Filtered runs remain +`executable-smoke` evidence and can never be promoted as powered effectiveness. +Every negative, invalid, or interrupted attempt remains in its append-only +attempt ledger. + +`npm run eval:ucr:qualification:verify` verifies hashes, signatures, selected +rows, and recomputed verdicts even when the recorded scientific outcome is a +failure. Release automation that intentionally requires at least one passing +active qualification uses `npm run eval:ucr:qualification:gate`. Keeping those +conditions separate prevents CI from incentivizing favorable-only reruns. + +The August 10 live program preserves every attempt rather than selecting only +favorable reruns. Earlier results are retained under `results/superseded/` after +each harness contract change. The final active artifacts share source-tree hash +`144ac09bc62567ac67c946c9afd811349c75c65cd6432efb54fb018d6737db21`. +In the integrity-valid adversarial Codex-to-Claude pair, both arms were correct, +runtime delivery occurred before the consumer, all six declared repository +anchors were verified, both provider models were exactly attested with zero +reroutes, and the consumer saw zero MCP tools. Runtime nevertheless used 17,482 +tokens versus 16,589 (+5.383%) and 55.820 seconds versus 39.377 (+41.758%), so +it failed the preregistered 5% token gate. The reciprocal control was valid and +correct, but the runtime Claude producer exhausted provider quota, so capture +was withheld and the pair failed closed. Antigravity 1.1.11 launched through +its registered transport but required account authentication; no Gemini model +or usage was claimed. These are signed bounded outcomes, not powered +effectiveness evidence. + +Every bounded execution emits a separate `ucr.study-qualification/1` verdict. +That gate requires complete selected execution, signed trial integrity, +independent grading, no severe harm, correct treatment and safety arms, +pre-action runtime delivery, control and hard-negative withholding, paired +coverage, correctness non-inferiority, and at most 5% observed token overhead. +An empty control may fail because prevention of that failure is a primary study +outcome; changing prompts, graders, powered thresholds, or source after the +gate passes invalidates scale-up. + +No direction passes the final hardened qualification contract. The full +54,054-trial / 113,022-call powered study therefore remains blocked rather than +spending the matrix budget on a design whose prerequisite directions fail. + +After every configured direction passes qualification without changing prompts, +graders, thresholds, or source, run the frozen powered study: + +```text +UCR_STUDY_SECRET= npm run eval:ucr:full-study +``` + +Promotable effectiveness, superiority, and production ledgers must use the same +externally provisioned Ed25519 identity. Configure +`UCR_EVIDENCE_SIGNING_KEY_ID`, `UCR_EVIDENCE_PRIVATE_KEY_FILE`, and +`UCR_EVIDENCE_PUBLIC_KEY_FILE`; artifacts retain only the published key id, not +either key. Production assembly additionally requires +`UCR_TRAFFIC_PSEUDONYM_SECRET` and `UCR_TRAFFIC_PSEUDONYM_KEY_ID`, and persists +only stable pseudonyms for client and project identifiers. + +The effectiveness verdict additionally requires all-family and all-arm coverage, +direction- and family-level correctness non-inferiority, direction-level token +non-inferiority, multiplicity-adjusted confidence-bound retrieval safety, a complete causal chain in +every family, complete trial integrity, independent grading, zero consumer MCP +schema, and zero capture-only inference. + +## 2. Competitive superiority + +`npm run eval:ucr:competitive:plan` audits the registry. Internal reference +algorithms are controls but cannot support a product claim. A promotable registry +must contain live, pinned, reproducible configurations for no memory, full +history, static instructions, vector RAG, graph RAG, memory OS, and vendor-native +memory. Product categories require a named product and published configuration. + +`npm run eval:ucr:competitive` reuses the exact effectiveness tasks, models, +permissions, context/retry/tool budgets, and independent graders. Every baseline +is repeated at least twice. UCR must improve correctness by more than ten points, +the paired confidence lower bound must exceed zero, and UCR must remain on the +correctness/harm/tokens/latency Pareto frontier for every required baseline. + +## 3. Production readiness + +`npm run eval:ucr:production:traffic:plan` prints the privacy-safe JSONL contract. +Local fault exercises never count as production traffic. Promotion requires at +least 1,000 opt-in samples across at least three clients and three projects over +at least seven days, covering shadow, observe-only, advisory canary, verification +canary, and scoped enforcement. It also requires passed SLOs, every fault +exercise, zero-loss recovery, zero privacy violations, zero severe harm, and the +stable rollout stage. + +The production assembler rejects prompt, transcript, or raw-output fields, +creates a signed production ledger, and keeps rollback/readiness separate from +effectiveness and superiority. + +## CI and dashboard + +`npm run verify:ucr:study-design` fails CI if any release metric loses its study, +any family/arm/client/mode disappears, or either the paired or hard-negative +sample size becomes underpowered. The dashboard renders effectiveness, +superiority, and production separately, along with missing metrics, family/arm +coverage, the worst hard-negative confidence upper bound, and the worst +directional token bound. + +No documentation or dashboard may call UCR a RAG/memory replacement until all +three signed verdicts pass. Negative and incomplete runs are evidence, not +artifacts to delete. diff --git a/evals/ucr/README.md b/evals/ucr/README.md index 66caa1b16..1bbc1ccaf 100644 --- a/evals/ucr/README.md +++ b/evals/ucr/README.md @@ -65,14 +65,17 @@ does not trust the model's final text. The stateful primary edges Codex→Claude Code and Claude Code→Codex pass with mandatory delivery, zero consumer MCP tools, and zero repeated predecessor -mistakes. In the latest source-complete runs, Codex→Claude increased total token -traffic 22.0% and latency 109.6%; Claude→Codex increased traffic 56.9% and -latency 91.9%. An earlier Claude→Codex attempt had improved both metrics, so the -directional reversal is explicit variance evidence, not a result to average -away. The efficiency gate is currently 0/2. The Copilot edge is retained as -negative evidence because its consumer invocation returned HTTP 402 after the -account quota was exhausted. These are executable smokes, not powered -effectiveness estimates. +mistakes. In the stricter matched rerun, Codex→Claude reduced total traffic +11.56% while latency increased 10.87%; Claude→Codex reduced traffic 0.58% and +latency 0.53%. Both controls were correct, so these rows establish executable +handoff and accounting—not a correctness or mistake-prevention effect. +That direction-level loss is not averaged away. The live harness now gives both +arms byte-identical predecessor prompts and the same native guard transport so +semantic-authoring and plugin startup are not treatment-only confounders; new +live evidence must be collected before the directional result can change. The +Copilot edge is retained as negative evidence because its consumer invocation +returned HTTP 402 after the account quota was exhausted. These are executable +smokes, not powered effectiveness estimates. A later two-direction pilot deliberately repeated the OpenAI↔Anthropic study: Codex→Claude passed and Claude→Codex did not deliver the correction, so the @@ -80,17 +83,26 @@ pilot passed 1/2. Its signed four-row ledger is included as an integrity-valid negative artifact. This variance is why the successful edges are not promoted to an effectiveness claim. -The powered handoff executor is preregistered separately: +The legacy powered handoff executor remains a direction-specific qualification +tool. It cannot establish all-family effectiveness. The replacement-grade study +is preregistered separately: ```sh -npm run eval:ucr:effectiveness:plan -npm run eval:ucr:effectiveness +npm run eval:ucr:full-study:powered-plan +npm run verify:ucr:study-design ``` -The frozen power analysis requires 357 pairs per arm in both directions, or -2,142 model invocations. Incomplete runs and smaller pilots are cryptographically -sealed as `executable-smoke`; the runner promotes a ledger to `effectiveness` -only after every preregistered pair completes. +The full plan covers all 11 families, seven arms, three independent model +families, and nine same/cross-client directions. It requires 363 paired +empty/runtime observations and 1,056 hard-negative opportunities per direction +and arm: 54,054 trial envelopes and 113,022 provider calls. Concurrent-agent +trials use one producer plus two overlapping successors instead of being labels. +Hard-negative and subgroup intervals control family-wise error across the +preregistered direction/arm comparisons. +The hard-negative count powers the confidence bound, not merely a zero-error +point estimate. Incomplete or smaller runs are sealed as `executable-smoke`. +See [`FULL_STUDY_CONTRACT.md`](./FULL_STUDY_CONTRACT.md) for the CLI driver, +competitive, production, causal, privacy, and promotion contracts. ## Release boundary @@ -100,3 +112,7 @@ are not present, so the project does not claim statistical efficiency, superiority over memory/RAG products, or production readiness. Missing usage is published as `null`; deterministic fixture labels are never represented as live model runs. + +CI now proves that every release metric has an executable study source. That is +design coverage, not measured effectiveness, and does not change the signed +`insufficient` verdict. diff --git a/evals/ucr/effectiveness-roadmap.md b/evals/ucr/effectiveness-roadmap.md index 9a0e908d6..8d3f98038 100644 --- a/evals/ucr/effectiveness-roadmap.md +++ b/evals/ucr/effectiveness-roadmap.md @@ -18,12 +18,14 @@ exit gate all pass. Signed negative runs remain in the evidence index. in-turn model-authored capture predecessor, a blinded control successor, and a runtime successor. Both fresh successors produced correct repository state and neither runtime successor repeated the generated-only edit. -- Across those two executable-smoke pairs, full predecessor-to-successor token - traffic fell from 879,257 to 694,125 (21.06%) and latency fell 18.36%. The - blinded controls repeated one known mistake while runtime repeated zero. In - Claude-to-Codex, a signed native Codex pre-tool guard denied the bad edit - before execution. Directionally, Claude-to-Codex still used 1.48% more token - traffic, so the aggregate point estimate is not a powered efficiency claim. +- The stricter matched rerun gives control and treatment producers byte-identical + prompts and equivalent client surfaces, and loads identical native guard + transport in both consumer arms. Full pipeline traffic fell from 702,243 to + 662,510 (5.66%), while latency rose from 146,289 ms to 153,210 ms (4.73%). + Codex-to-Claude used 11.56% fewer tokens but 10.87% more latency; + Claude-to-Codex used 0.58% fewer tokens and 0.53% less latency. Both controls + completed correctly without the target mistake, so the rerun is not evidence + of a correctness lift or mistake prevention. - A minimal Claude plugin transport attempt increased traffic 25.0% and latency 25.3% without exercising the guard. It remains negative evidence and caused the harness to use a one-hook native settings file instead of loading the @@ -31,6 +33,12 @@ exit gate all pass. Signed negative runs remain in the evidence index. - Claude-to-Copilot reached the consumer step but the installed account returned HTTP 402 for exhausted quota. The failed signed row is retained; Copilot is not live-certified by this run. +- The final model-transparent adversarial pair exactly attested Codex + `gpt-5.6-sol` and Claude Sonnet 5 with zero reroutes and zero consumer MCP + exposure. Both arms were correct, but runtime increased tokens 5.383% and + latency 41.758%, so the direction failed qualification. The reciprocal + runtime producer hit Claude quota; Antigravity required authentication. The + powered matrix remains blocked and no earlier favorable rerun is promoted. ## Non-negotiable runtime contract @@ -48,29 +56,34 @@ responsible for remembering to retrieve prerequisite context. ## All 18 workstreams -| # | Workstream | Current state | Required exit | -| --- | --- | --- | --- | -| 1 | Universal Cognitive Event Protocol | Mechanism implemented | Version/migration replay, signatures, redaction allowlist, and independent verifier all pass on every promoted artifact | -| 2 | Typed temporal and causal graph | Conformance implemented | Million-event integrity plus adversarial malformed, duplicate, causal-cycle, and bounded-resource runs pass without overstating memory guarantees | -| 3 | Model-native semantic compiler | Strengthened in this PR | Active model authors exact persisted bytes only after host-authenticated evidence; malformed output retries once; poisoned or unverifiable semantics never activate | -| 4 | Executable memory and action guards | Shared runtime, ten native adapters, and one live denial implemented | Verified corrections compile into scoped pre-tool guards; every interceptable client blocks or redirects the repeated action before execution; advisory clients are labeled | -| 5 | Causal credit and harm quarantine | Deterministic mechanism implemented | Blinded ablations attribute benefit without confounders; severe harm is quarantined before reuse; false-credit and quarantine-latency gates pass | -| 6 | Context virtual machine | One-capsule delivery and phase accounting implemented; powered traffic gate open | Report added active context separately from total traffic, cached traffic, turns, latency, and cost; p95 added context stays under the preregistered budget and total traffic is non-inferior | -| 7 | Complete checkpoints and cross-model takeover | Stateful primary smoke passes | Ten consecutive fresh-session pairs per primary direction pass compatibility, delivery, end-state correctness, zero-repeat, and efficiency gates | -| 8 | Distributed multi-agent coordination | Physical-process conformance implemented | Multiple real agents concurrently publish, lease, conflict, recover, and take over the same task with no lost accepted writes or duplicate work | -| 9 | Autonomous consolidation and forgetting | Deterministic mechanism implemented | Long-horizon live reuse retains delayed-value cognition, removes stale/low-value items, bounds growth, and never mutates source evidence | -| 10 | Secure cross-project federation | Policy conformance implemented | Cross-project opt-in, principals, taint, redaction, revocation, and scope isolation pass adversarial live transfer with zero unauthorized delivery | -| 11 | Adaptive retrieval | Exact scope kernel and fallback hardened | Hidden hard-negative suite reaches <1% irrelevant/stale delivery with calibrated abstention and no empty calibrated route disabling retrieval | -| 12 | Minimal MCP capability surface | Measured and strengthened | Normal consumers expose zero tools; continuity is 480 tokens, optional attestation 71, extended cognitive 694, core 4,815, and full 30,125; provider wrappers are measured separately | -| 13 | All-client adapter SDK | Ten generated native guard adapters pass conformance; two primary live edges pass | Every supported CLI is certified against the same contract, with task/pre-tool guarantees stated per lifecycle family and paid live edges for available providers | -| 14 | Cognitive Continuity Benchmark | Stateful runner implemented | Natural tasks grade executable end state rather than answer echoes; hidden variants, signed graders, contamination checks, and preregistration all pass | -| 15 | Competitive baseline harness | Deterministic harness implemented; product runs open | AGENTS.md, transcript, RAG/vector, and product-memory arms use identical tasks, models, budgets, tool access, and independent graders | -| 16 | Compounding-learning experiments | Deterministic study only | Longitudinal live model study shows increasing first-pass correctness and decreasing repeated errors/context traffic with confidence intervals, not fixture-only gains | -| 17 | Effectiveness dashboard and release gates | Dashboard and fail-closed tiers implemented | Dashboard displays signed positive and negative stateful directions, active-context versus traffic metrics, confidence intervals, and refuses effectiveness/superiority promotion until gates pass | -| 18 | Shadow/canary/production hardening | Fault mechanism exercise only | Real shadow/canary window meets SLOs with scoped kill switch, rollback, breaker recovery, immutable incidents, privacy review, and no severe unquarantined harm | +| # | Workstream | Current state | Required exit | +| --- | --------------------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | Universal Cognitive Event Protocol | Mechanism implemented | Version/migration replay, signatures, redaction allowlist, and independent verifier all pass on every promoted artifact | +| 2 | Typed temporal and causal graph | Conformance implemented | Million-event integrity plus adversarial malformed, duplicate, causal-cycle, and bounded-resource runs pass without overstating memory guarantees | +| 3 | Model-native semantic compiler | Strengthened in this PR | Active model authors exact persisted bytes only after host-authenticated evidence; malformed output retries once; poisoned or unverifiable semantics never activate | +| 4 | Executable memory and action guards | Shared runtime, ten native adapters, and one live denial implemented | Verified corrections compile into scoped pre-tool guards; every interceptable client blocks or redirects the repeated action before execution; advisory clients are labeled | +| 5 | Causal credit and harm quarantine | Deterministic mechanism implemented | Blinded ablations attribute benefit without confounders; severe harm is quarantined before reuse; false-credit and quarantine-latency gates pass | +| 6 | Context virtual machine | One-capsule delivery and phase accounting implemented; powered traffic gate open | Report added active context separately from total traffic, cached traffic, turns, latency, and cost; p95 added context stays under the preregistered budget and total traffic is non-inferior | +| 7 | Complete checkpoints and cross-model takeover | Stateful primary smoke passes | Ten consecutive fresh-session pairs per primary direction pass compatibility, delivery, end-state correctness, zero-repeat, and efficiency gates | +| 8 | Distributed multi-agent coordination | Physical-process conformance implemented | Multiple real agents concurrently publish, lease, conflict, recover, and take over the same task with no lost accepted writes or duplicate work | +| 9 | Autonomous consolidation and forgetting | Deterministic mechanism implemented | Long-horizon live reuse retains delayed-value cognition, removes stale/low-value items, bounds growth, and never mutates source evidence | +| 10 | Secure cross-project federation | Policy conformance implemented | Cross-project opt-in, principals, taint, redaction, revocation, and scope isolation pass adversarial live transfer with zero unauthorized delivery | +| 11 | Adaptive retrieval | Exact scope kernel and fallback hardened | Hidden hard-negative suite reaches <1% irrelevant/stale delivery with calibrated abstention and no empty calibrated route disabling retrieval | +| 12 | Minimal MCP capability surface | Measured and strengthened | Normal consumers expose zero tools; continuity is 480 tokens, optional attestation 71, extended cognitive 694, core 4,815, and full 30,125; provider wrappers are measured separately | +| 13 | All-client adapter SDK | Ten generated native guard adapters pass conformance; two primary live edges pass | Every supported CLI is certified against the same contract, with task/pre-tool guarantees stated per lifecycle family and paid live edges for available providers | +| 14 | Cognitive Continuity Benchmark | Stateful runner implemented | Natural tasks grade executable end state rather than answer echoes; hidden variants, signed graders, contamination checks, and preregistration all pass | +| 15 | Competitive baseline harness | Deterministic harness implemented; product runs open | AGENTS.md, transcript, RAG/vector, and product-memory arms use identical tasks, models, budgets, tool access, and independent graders | +| 16 | Compounding-learning experiments | Deterministic study only | Longitudinal live model study shows increasing first-pass correctness and decreasing repeated errors/context traffic with confidence intervals, not fixture-only gains | +| 17 | Effectiveness dashboard and release gates | Dashboard and fail-closed tiers implemented | Dashboard displays signed positive and negative stateful directions, active-context versus traffic metrics, confidence intervals, and refuses effectiveness/superiority promotion until gates pass | +| 18 | Shadow/canary/production hardening | Fault mechanism exercise only | Real shadow/canary window meets SLOs with scoped kill switch, rollback, breaker recovery, immutable incidents, privacy review, and no severe unquarantined harm | ## Execution sequence +The authoritative executable design is now +[`FULL_STUDY_CONTRACT.md`](./FULL_STUDY_CONTRACT.md). It supersedes the earlier +single-fixture powered estimate below: repeating one handoff task cannot establish +all-family generalization or confidence-bounded hard-negative safety. + 1. Ship the mandatory `PreActionController`, exact-scope retrieval, model-authored harvest receipts, zero-tool consumer path, capability guarantees, and the stateful bidirectional evidence now implemented on this branch. @@ -79,14 +92,15 @@ responsible for remembering to retrieve prerequisite context. on every interceptable lifecycle family. 3. Keep the adversarial retrieval/poisoning/scope suites and phase-scoped accounting as release gates for maximum active context, total traffic, cache - traffic, model turns, latency, and cost. Investigate the remaining - Claude-to-Codex directional token regression before promotion. + traffic, model turns, latency, and cost. Investigate the Codex-to-Claude + latency regression before promotion. 4. Run ten consecutive stateful pairs in every available direction. Acquire or replenish provider quota before calling Copilot or unavailable clients live; never substitute a deterministic adapter test for a paid live edge. -5. Freeze graders and hidden variants, then execute the preregistered powered - paired study (357 pairs per direction, 2,856 calls for two directions under - the four-call protocol). Require at +5. Freeze graders and hidden variants, then execute the full preregistered matrix + (363 pairs per direction and 1,056 hard-negative opportunities per direction + and arm; 54,054 trials / 113,022 provider calls for three clients and + nine same/cross-client directions). Require at least a 10-point correctness lift with confidence-interval lower bound above zero, zero severe unquarantined harm, and the efficiency SLO. 6. Run fair competitive baselines, then shadow/canary production. Only those diff --git a/evals/ucr/results/deterministic-verification-v1.json b/evals/ucr/results/deterministic-verification-v1.json index 27dd7c120..7e1dec870 100644 --- a/evals/ucr/results/deterministic-verification-v1.json +++ b/evals/ucr/results/deterministic-verification-v1.json @@ -46,7 +46,7 @@ "modelRuns": 9, "executableClientSmokes": 3, "competitiveReproductions": 0, - "reportHash": "fe648170e0a37e83367ad1847221c6969fa66997532e38cbb22758178ce3ad2e", + "reportHash": "862391eee20caae26ff445ea3f5ae0fc71eb5dbdd453deebb183c775293ac3e8", "evidenceTiers": { "transport": { "status": "missing", @@ -57,23 +57,26 @@ "conformance": { "status": "present", "ledgers": 2, - "rows": 706, + "rows": 707, "hashes": [ - "7ada2695725642896ae1f90cc30bc6c3b8b2232787ddc6c7cd16ef8eb268e217", + "83fad7c4304136839c9d40a6cd978a3098b5e580dd3ac4bec7d71b38f00e8c56", "7d1f5cf0f9b027c2a6e3130860a17e580b6989ac2f6776237858ab296408545a" ] }, "executable-smoke": { "status": "present", - "ledgers": 6, - "rows": 14, + "ledgers": 9, + "rows": 19, "hashes": [ "96000a35253e6c02ec6d0937cca0fde20fc2e080c19e901e3b13d2ffeb9fa8b9", - "1d0533340a4a4a1387ebde702bfb8c9b0903ea9ce38b7eea1ac18b79069dd7f7", - "88acebf93343ce71b5adc79bcbc160e0f36b081d3c3fce9d29b1324c53864a4c", + "0efd8b4dcaa975017b0509c47906b34d7b836d45f3d3e976be787fd4ea2a2c41", + "b8c8371750b77e12f60507be373aedbca433ecabf7c9a29ba3ecf842ad07822d", "96fb850199d15ac70aebd837075b1fdae61214b3c88b293681679be8c558e999", "f4383a405fd5fec280108b65506179c78a8418a905ba1643277dc75b0478aef7", - "ceac3e14739ed4b580f725befd832b9b3bce66718f451a1096cbcf7f6db77711" + "ceac3e14739ed4b580f725befd832b9b3bce66718f451a1096cbcf7f6db77711", + "e1b9795d36ae9a41c50f6ebc43910e72f0113c798bf69e6aa489bf57eb1591a6", + "34351afe3c4d9763cac84ebeb2d9ecd8a2dcf2142064c97c8ae9d69036fe0169", + "42dbeba5e30bb4f1f138edc0d344560b8494217dd20a36a411d5d871b76d79c7" ] }, "effectiveness": { @@ -102,8 +105,12 @@ "passed": false, "missing": [ "applicabilityPrecision", + "applicabilityPrecisionIntervalLow", "preActionDelivery", + "preActionDeliveryIntervalLow", + "irrelevantDeliveryIntervalHigh", "staleDelivery", + "staleDeliveryIntervalHigh", "recurrenceReduction", "recurrenceIntervalLow", "naturalCorrectnessDelta", @@ -116,10 +123,22 @@ "latencyOverheadP95", "knownMistakeRecurrence", "contradictoryDelivery", + "contradictoryDeliveryIntervalHigh", + "negativeDeliveryIntervalHigh", "consumerSchemaTokensP95", "captureModelCallsP95", "crossClientPassed", - "competitivePassed" + "benchmarkFamilyCoverage", + "benchmarkArmCoverage", + "directionalCorrectnessIntervalLow", + "familyCorrectnessIntervalLow", + "directionalTokenOverheadHigh", + "causalChainIntegrity", + "trialIntegrity", + "independentGrading", + "competitivePassed", + "competitiveCoverage", + "competitiveReproducibility" ], "failed": [] }, @@ -152,14 +171,14 @@ { "name": "unified signed evidence index is integrity-valid", "passed": true, - "detail": "12/12 artifacts" + "detail": "16/16 artifacts" }, { "name": "stateful cross-model smoke passes both primary directions and retains the blocked edge", "passed": true, "detail": { - "artifactsValid": 12, - "artifactsTotal": 12, + "artifactsValid": 16, + "artifactsTotal": 16, "liveDirectionsPassed": 2, "liveDirectionsAttempted": 3, "liveDirectionMetrics": [ @@ -178,13 +197,13 @@ "nativeGuardEnforced": false, "captureModelCalls": 0, "consumerStaticSchemaTokens": 0, - "capsuleTokens": 64, - "controlTokenTraffic": 476301, - "runtimeTokenTraffic": 285194, - "tokenTrafficDelta": -0.4012315741516394, - "controlLatencyMs": 73503.72200000001, - "runtimeLatencyMs": 66832.0761, - "latencyDelta": -0.09076609617129323 + "capsuleTokens": 57, + "controlTokenTraffic": 324886, + "runtimeTokenTraffic": 287344, + "tokenTrafficDelta": -0.11555437907450614, + "controlLatencyMs": 67516.1478, + "runtimeLatencyMs": 74853.2538, + "latencyDelta": 0.10867186945757588 }, { "direction": "claude-code->codex", @@ -195,19 +214,19 @@ "predecessorMistakeObserved": true, "predecessorCorrectionVerified": true, "repeatedFailure": false, - "controlMistake": true, + "controlMistake": false, "delivered": true, "nativeGuardWired": true, - "nativeGuardEnforced": true, + "nativeGuardEnforced": false, "captureModelCalls": 0, "consumerStaticSchemaTokens": 0, - "capsuleTokens": 91, - "controlTokenTraffic": 402956, - "runtimeTokenTraffic": 408931, - "tokenTrafficDelta": 0.01482792165894043, - "controlLatencyMs": 115158.38889999998, - "runtimeLatencyMs": 87188.1611, - "latencyDelta": -0.24288484813979527 + "capsuleTokens": 98, + "controlTokenTraffic": 377357, + "runtimeTokenTraffic": 375166, + "tokenTrafficDelta": -0.005806172934383091, + "controlLatencyMs": 78773.0749, + "runtimeLatencyMs": 78356.7587, + "latencyDelta": -0.00528500633660044 }, { "direction": "claude-code->copilot", @@ -233,13 +252,13 @@ "latencyDelta": -0.15731011708195736 } ], - "liveDirectionsWithLowerTokenTraffic": 1, - "liveDirectionsWithLowerLatency": 2, - "combinedLiveTokenReduction": 0.210555048182727, - "combinedLiveLatencyReduction": 0.183618605425028, - "blindedControlMistakes": 1, + "liveDirectionsWithLowerTokenTraffic": 2, + "liveDirectionsWithLowerLatency": 1, + "combinedLiveTokenReduction": 0.05658012967021387, + "combinedLiveLatencyReduction": -0.047308951898614456, + "blindedControlMistakes": 0, "runtimeKnownMistakeRecurrences": 0, - "nativeGuardEnforcements": 1, + "nativeGuardEnforcements": 0, "maximumCaptureModelCalls": 0, "maximumConsumerStaticSchemaTokens": 0, "registeredClientProcesses": 16, @@ -247,15 +266,35 @@ "coordinationWorkers": 100, "productionFaults": 6, "cognitiveSchemaTokens": 480, - "cognitiveReductionVsFull": 0.9840663900414938 + "cognitiveReductionVsFull": 0.9840663900414938, + "studyDesign": { + "passed": true, + "trials": 54054, + "providerInvocations": 113022, + "mappedMetrics": 37, + "representativeStudyClients": 3, + "universalDriverClients": 16, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 1420 + } + } } }, { "name": "scale, coordination, adapter, and fault artifacts meet frozen targets", "passed": true, "detail": { - "artifactsValid": 12, - "artifactsTotal": 12, + "artifactsValid": 16, + "artifactsTotal": 16, "liveDirectionsPassed": 2, "liveDirectionsAttempted": 3, "liveDirectionMetrics": [ @@ -274,13 +313,13 @@ "nativeGuardEnforced": false, "captureModelCalls": 0, "consumerStaticSchemaTokens": 0, - "capsuleTokens": 64, - "controlTokenTraffic": 476301, - "runtimeTokenTraffic": 285194, - "tokenTrafficDelta": -0.4012315741516394, - "controlLatencyMs": 73503.72200000001, - "runtimeLatencyMs": 66832.0761, - "latencyDelta": -0.09076609617129323 + "capsuleTokens": 57, + "controlTokenTraffic": 324886, + "runtimeTokenTraffic": 287344, + "tokenTrafficDelta": -0.11555437907450614, + "controlLatencyMs": 67516.1478, + "runtimeLatencyMs": 74853.2538, + "latencyDelta": 0.10867186945757588 }, { "direction": "claude-code->codex", @@ -291,19 +330,19 @@ "predecessorMistakeObserved": true, "predecessorCorrectionVerified": true, "repeatedFailure": false, - "controlMistake": true, + "controlMistake": false, "delivered": true, "nativeGuardWired": true, - "nativeGuardEnforced": true, + "nativeGuardEnforced": false, "captureModelCalls": 0, "consumerStaticSchemaTokens": 0, - "capsuleTokens": 91, - "controlTokenTraffic": 402956, - "runtimeTokenTraffic": 408931, - "tokenTrafficDelta": 0.01482792165894043, - "controlLatencyMs": 115158.38889999998, - "runtimeLatencyMs": 87188.1611, - "latencyDelta": -0.24288484813979527 + "capsuleTokens": 98, + "controlTokenTraffic": 377357, + "runtimeTokenTraffic": 375166, + "tokenTrafficDelta": -0.005806172934383091, + "controlLatencyMs": 78773.0749, + "runtimeLatencyMs": 78356.7587, + "latencyDelta": -0.00528500633660044 }, { "direction": "claude-code->copilot", @@ -329,13 +368,13 @@ "latencyDelta": -0.15731011708195736 } ], - "liveDirectionsWithLowerTokenTraffic": 1, - "liveDirectionsWithLowerLatency": 2, - "combinedLiveTokenReduction": 0.210555048182727, - "combinedLiveLatencyReduction": 0.183618605425028, - "blindedControlMistakes": 1, + "liveDirectionsWithLowerTokenTraffic": 2, + "liveDirectionsWithLowerLatency": 1, + "combinedLiveTokenReduction": 0.05658012967021387, + "combinedLiveLatencyReduction": -0.047308951898614456, + "blindedControlMistakes": 0, "runtimeKnownMistakeRecurrences": 0, - "nativeGuardEnforcements": 1, + "nativeGuardEnforcements": 0, "maximumCaptureModelCalls": 0, "maximumConsumerStaticSchemaTokens": 0, "registeredClientProcesses": 16, @@ -343,7 +382,27 @@ "coordinationWorkers": 100, "productionFaults": 6, "cognitiveSchemaTokens": 480, - "cognitiveReductionVsFull": 0.9840663900414938 + "cognitiveReductionVsFull": 0.9840663900414938, + "studyDesign": { + "passed": true, + "trials": 54054, + "providerInvocations": 113022, + "mappedMetrics": 37, + "representativeStudyClients": 3, + "universalDriverClients": 16, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 1420 + } + } } }, { @@ -507,7 +566,7 @@ { "name": "release gate refuses to infer effectiveness from deterministic conformance", "passed": true, - "detail": "19 live fields missing" + "detail": "35 live fields missing" }, { "name": "canary regression automatically rolls back and scopes a kill switch", @@ -525,5 +584,5 @@ } } ], - "reportHash": "6068ad7ff40be7be4c14a16bd80e3d04cc0f5f4f7cafc8232bc8ba8ab92175b7" + "reportHash": "eda1fc7e69eece53eac45b224c9863e9498b22d7a9a20f3ac291a5c1bce9858e" } diff --git a/evals/ucr/results/evidence-index-v2.json b/evals/ucr/results/evidence-index-v2.json index fa422bb55..4cdd232f6 100644 --- a/evals/ucr/results/evidence-index-v2.json +++ b/evals/ucr/results/evidence-index-v2.json @@ -1,6 +1,6 @@ { "schemaVersion": "ucr.evidence-index/2", - "generatedAt": "2026-08-10T04:54:55.796Z", + "generatedAt": "2026-08-10T19:16:42.192Z", "evidenceContract": { "claimPolicy": "metrics are derived only from ledgers at or above each metric evidence requirement", "tiers": { @@ -13,23 +13,26 @@ "conformance": { "status": "present", "ledgers": 2, - "rows": 706, + "rows": 707, "hashes": [ - "7ada2695725642896ae1f90cc30bc6c3b8b2232787ddc6c7cd16ef8eb268e217", + "83fad7c4304136839c9d40a6cd978a3098b5e580dd3ac4bec7d71b38f00e8c56", "7d1f5cf0f9b027c2a6e3130860a17e580b6989ac2f6776237858ab296408545a" ] }, "executable-smoke": { "status": "present", - "ledgers": 6, - "rows": 14, + "ledgers": 9, + "rows": 19, "hashes": [ "96000a35253e6c02ec6d0937cca0fde20fc2e080c19e901e3b13d2ffeb9fa8b9", - "1d0533340a4a4a1387ebde702bfb8c9b0903ea9ce38b7eea1ac18b79069dd7f7", - "88acebf93343ce71b5adc79bcbc160e0f36b081d3c3fce9d29b1324c53864a4c", + "0efd8b4dcaa975017b0509c47906b34d7b836d45f3d3e976be787fd4ea2a2c41", + "b8c8371750b77e12f60507be373aedbca433ecabf7c9a29ba3ecf842ad07822d", "96fb850199d15ac70aebd837075b1fdae61214b3c88b293681679be8c558e999", "f4383a405fd5fec280108b65506179c78a8418a905ba1643277dc75b0478aef7", - "ceac3e14739ed4b580f725befd832b9b3bce66718f451a1096cbcf7f6db77711" + "ceac3e14739ed4b580f725befd832b9b3bce66718f451a1096cbcf7f6db77711", + "e1b9795d36ae9a41c50f6ebc43910e72f0113c798bf69e6aa489bf57eb1591a6", + "34351afe3c4d9763cac84ebeb2d9ecd8a2dcf2142064c97c8ae9d69036fe0169", + "42dbeba5e30bb4f1f138edc0d344560b8494217dd20a36a411d5d871b76d79c7" ] }, "effectiveness": { @@ -53,6 +56,23 @@ } }, "artifacts": [ + { + "name": "study-readiness", + "filename": "study-readiness-v1.json", + "evidenceClass": "conformance", + "requiredPass": true, + "present": true, + "valid": true, + "reportHashValid": true, + "reportHash": "02488e8c1f6d1b4495bdc14027778946ff69894e7709808b25d89f3e4235ace1", + "schemaVersion": "ucr.study-readiness/1", + "passed": true, + "executedAt": null, + "sourceTreeHash": null, + "ledgerHash": null, + "ledgerVerification": null, + "keyResolutionError": null + }, { "name": "graph-scale", "filename": "graph-scale-v1.json", @@ -67,7 +87,8 @@ "executedAt": "2026-08-09T22:14:45.579Z", "sourceTreeHash": "5f0a584fed1e8c233b262296a089ca4236edd4b6f616b13418714989ae32a8d6", "ledgerHash": null, - "ledgerVerification": null + "ledgerVerification": null, + "keyResolutionError": null }, { "name": "coordination-scale", @@ -83,7 +104,8 @@ "executedAt": "2026-08-09T22:23:49.552Z", "sourceTreeHash": "ea29500316b80d8806095cff65bd3ece6e4682a9d574c6bf3c6d02f6c0e18e58", "ledgerHash": null, - "ledgerVerification": null + "ledgerVerification": null, + "keyResolutionError": null }, { "name": "consolidation", @@ -99,7 +121,8 @@ "executedAt": "2026-08-09T22:24:48.511Z", "sourceTreeHash": "a34a5ebd8aa6604f8291e4f39a3e6a50636579635cab2d4ec262cb78cfd266b2", "ledgerHash": null, - "ledgerVerification": null + "ledgerVerification": null, + "keyResolutionError": null }, { "name": "adapter-processes", @@ -115,7 +138,8 @@ "executedAt": "2026-08-10T04:42:26.916Z", "sourceTreeHash": "8a36da74f0111c1d4a252dc257fc1fe24d7f3b7a8057ce5488cec6b4a83e61d5", "ledgerHash": null, - "ledgerVerification": null + "ledgerVerification": null, + "keyResolutionError": null }, { "name": "mcp-context", @@ -131,7 +155,8 @@ "executedAt": "2026-08-10T04:42:33.220Z", "sourceTreeHash": null, "ledgerHash": null, - "ledgerVerification": null + "ledgerVerification": null, + "keyResolutionError": null }, { "name": "compounding", @@ -153,7 +178,8 @@ "validSignature": true, "ledgerHash": "7d1f5cf0f9b027c2a6e3130860a17e580b6989ac2f6776237858ab296408545a", "rowCount": 700 - } + }, + "keyResolutionError": null }, { "name": "production-exercise", @@ -175,51 +201,54 @@ "validSignature": true, "ledgerHash": "96000a35253e6c02ec6d0937cca0fde20fc2e080c19e901e3b13d2ffeb9fa8b9", "rowCount": 6 - } + }, + "keyResolutionError": null }, { "name": "stateful-codex-to-claude", - "filename": "stateful-preflight-codex-to-claude-v4.json", + "filename": "stateful-matched-codex-to-claude-v1.json", "evidenceClass": "executable-smoke", "requiredPass": true, "present": true, "valid": true, "reportHashValid": true, - "reportHash": "16f20a57c882fb91cf7eef90bf8a0daa92cfdbcb09f177e9aadd71a31fc9bda7", + "reportHash": "7b2b74545eed9ef9eb78dbc9dab6ba23f7e56a1cf121473830e4a3a28ad9695e", "schemaVersion": "ucr.live-multimodel-handoff/1", "passed": true, - "executedAt": "2026-08-10T04:26:46.370Z", - "sourceTreeHash": "3eba9476a6ad29ff4ddaa3a91b15846a1e7c95779157bee7927e4ff1a4372828", - "ledgerHash": "1d0533340a4a4a1387ebde702bfb8c9b0903ea9ce38b7eea1ac18b79069dd7f7", + "executedAt": "2026-08-10T12:39:01.830Z", + "sourceTreeHash": "96fcf325cd1945645579e83c48f0164c569017fdf952546ebe174f225c24df6d", + "ledgerHash": "0efd8b4dcaa975017b0509c47906b34d7b836d45f3d3e976be787fd4ea2a2c41", "ledgerVerification": { "valid": true, "diagnostics": [], "validSignature": true, - "ledgerHash": "1d0533340a4a4a1387ebde702bfb8c9b0903ea9ce38b7eea1ac18b79069dd7f7", + "ledgerHash": "0efd8b4dcaa975017b0509c47906b34d7b836d45f3d3e976be787fd4ea2a2c41", "rowCount": 1 - } + }, + "keyResolutionError": null }, { "name": "stateful-claude-to-codex", - "filename": "stateful-preflight-claude-to-codex-v3.json", + "filename": "stateful-matched-claude-to-codex-v1.json", "evidenceClass": "executable-smoke", "requiredPass": true, "present": true, "valid": true, "reportHashValid": true, - "reportHash": "0d01eb4eb9f21ddb5c590095c270ef4564267e856afe01fcd49d16cb0f3603aa", + "reportHash": "0fe914b29aece48d91c10f6a23c7b09e02b79f298e01829cafe0c9ab729d241e", "schemaVersion": "ucr.live-multimodel-handoff/1", "passed": true, - "executedAt": "2026-08-10T04:32:49.612Z", - "sourceTreeHash": "620eeab06b88a32a04b9a069f1b27c2451ff32bf5d3128790a3416b4a64dcdc6", - "ledgerHash": "88acebf93343ce71b5adc79bcbc160e0f36b081d3c3fce9d29b1324c53864a4c", + "executedAt": "2026-08-10T12:44:14.819Z", + "sourceTreeHash": "96fcf325cd1945645579e83c48f0164c569017fdf952546ebe174f225c24df6d", + "ledgerHash": "b8c8371750b77e12f60507be373aedbca433ecabf7c9a29ba3ecf842ad07822d", "ledgerVerification": { "valid": true, "diagnostics": [], "validSignature": true, - "ledgerHash": "88acebf93343ce71b5adc79bcbc160e0f36b081d3c3fce9d29b1324c53864a4c", + "ledgerHash": "b8c8371750b77e12f60507be373aedbca433ecabf7c9a29ba3ecf842ad07822d", "rowCount": 1 - } + }, + "keyResolutionError": null }, { "name": "transport-negative-claude-plugin", @@ -241,7 +270,8 @@ "validSignature": true, "ledgerHash": "96fb850199d15ac70aebd837075b1fdae61214b3c88b293681679be8c558e999", "rowCount": 1 - } + }, + "keyResolutionError": null }, { "name": "stateful-claude-to-copilot-quota", @@ -263,7 +293,8 @@ "validSignature": true, "ledgerHash": "f4383a405fd5fec280108b65506179c78a8418a905ba1643277dc75b0478aef7", "rowCount": 1 - } + }, + "keyResolutionError": null }, { "name": "effectiveness-pilot-negative", @@ -285,12 +316,82 @@ "validSignature": true, "ledgerHash": "ceac3e14739ed4b580f725befd832b9b3bce66718f451a1096cbcf7f6db77711", "rowCount": 4 - } + }, + "keyResolutionError": null + }, + { + "name": "full-study-blocked-codex-to-gemini-v4", + "filename": "full-study-blocked-codex-to-gemini-v4.json", + "evidenceClass": "blocked-smoke", + "requiredPass": false, + "present": true, + "valid": true, + "reportHashValid": true, + "reportHash": "a4c828e1fd8995a06ab953b3abe3da8ece2db279f6efb7800f0daa2ae55cb323", + "schemaVersion": "ucr.full-study/1", + "passed": false, + "executedAt": "2026-08-10T19:02:57.070Z", + "sourceTreeHash": null, + "ledgerHash": "e1b9795d36ae9a41c50f6ebc43910e72f0113c798bf69e6aa489bf57eb1591a6", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "e1b9795d36ae9a41c50f6ebc43910e72f0113c798bf69e6aa489bf57eb1591a6", + "rowCount": 1 + }, + "keyResolutionError": null + }, + { + "name": "full-study-qualification-claude-to-codex-v10", + "filename": "full-study-qualification-claude-to-codex-v10.json", + "evidenceClass": "executable-smoke", + "requiredPass": false, + "present": true, + "valid": true, + "reportHashValid": true, + "reportHash": "6138feb2a4b82d9ff3c2fa66760aef597cd1c7c88f5d785c3fb133e4e9fba4ec", + "schemaVersion": "ucr.full-study/1", + "passed": false, + "executedAt": "2026-08-10T19:00:07.732Z", + "sourceTreeHash": null, + "ledgerHash": "34351afe3c4d9763cac84ebeb2d9ecd8a2dcf2142064c97c8ae9d69036fe0169", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "34351afe3c4d9763cac84ebeb2d9ecd8a2dcf2142064c97c8ae9d69036fe0169", + "rowCount": 2 + }, + "keyResolutionError": null + }, + { + "name": "full-study-qualification-codex-to-claude-v10", + "filename": "full-study-qualification-codex-to-claude-v10.json", + "evidenceClass": "executable-smoke", + "requiredPass": false, + "present": true, + "valid": true, + "reportHashValid": true, + "reportHash": "ebf74f85edc65b9f52d6b86f2ac24048ba6c0a8e9ef4ab5f00fbc76ea32e1f54", + "schemaVersion": "ucr.full-study/1", + "passed": false, + "executedAt": "2026-08-10T18:54:33.874Z", + "sourceTreeHash": null, + "ledgerHash": "42dbeba5e30bb4f1f138edc0d344560b8494217dd20a36a411d5d871b76d79c7", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "42dbeba5e30bb4f1f138edc0d344560b8494217dd20a36a411d5d871b76d79c7", + "rowCount": 2 + }, + "keyResolutionError": null } ], "summary": { - "artifactsValid": 12, - "artifactsTotal": 12, + "artifactsValid": 16, + "artifactsTotal": 16, "liveDirectionsPassed": 2, "liveDirectionsAttempted": 3, "liveDirectionMetrics": [ @@ -309,13 +410,13 @@ "nativeGuardEnforced": false, "captureModelCalls": 0, "consumerStaticSchemaTokens": 0, - "capsuleTokens": 64, - "controlTokenTraffic": 476301, - "runtimeTokenTraffic": 285194, - "tokenTrafficDelta": -0.4012315741516394, - "controlLatencyMs": 73503.72200000001, - "runtimeLatencyMs": 66832.0761, - "latencyDelta": -0.09076609617129323 + "capsuleTokens": 57, + "controlTokenTraffic": 324886, + "runtimeTokenTraffic": 287344, + "tokenTrafficDelta": -0.11555437907450614, + "controlLatencyMs": 67516.1478, + "runtimeLatencyMs": 74853.2538, + "latencyDelta": 0.10867186945757588 }, { "direction": "claude-code->codex", @@ -326,19 +427,19 @@ "predecessorMistakeObserved": true, "predecessorCorrectionVerified": true, "repeatedFailure": false, - "controlMistake": true, + "controlMistake": false, "delivered": true, "nativeGuardWired": true, - "nativeGuardEnforced": true, + "nativeGuardEnforced": false, "captureModelCalls": 0, "consumerStaticSchemaTokens": 0, - "capsuleTokens": 91, - "controlTokenTraffic": 402956, - "runtimeTokenTraffic": 408931, - "tokenTrafficDelta": 0.01482792165894043, - "controlLatencyMs": 115158.38889999998, - "runtimeLatencyMs": 87188.1611, - "latencyDelta": -0.24288484813979527 + "capsuleTokens": 98, + "controlTokenTraffic": 377357, + "runtimeTokenTraffic": 375166, + "tokenTrafficDelta": -0.005806172934383091, + "controlLatencyMs": 78773.0749, + "runtimeLatencyMs": 78356.7587, + "latencyDelta": -0.00528500633660044 }, { "direction": "claude-code->copilot", @@ -364,13 +465,13 @@ "latencyDelta": -0.15731011708195736 } ], - "liveDirectionsWithLowerTokenTraffic": 1, - "liveDirectionsWithLowerLatency": 2, - "combinedLiveTokenReduction": 0.210555048182727, - "combinedLiveLatencyReduction": 0.183618605425028, - "blindedControlMistakes": 1, + "liveDirectionsWithLowerTokenTraffic": 2, + "liveDirectionsWithLowerLatency": 1, + "combinedLiveTokenReduction": 0.05658012967021387, + "combinedLiveLatencyReduction": -0.047308951898614456, + "blindedControlMistakes": 0, "runtimeKnownMistakeRecurrences": 0, - "nativeGuardEnforcements": 1, + "nativeGuardEnforcements": 0, "maximumCaptureModelCalls": 0, "maximumConsumerStaticSchemaTokens": 0, "registeredClientProcesses": 16, @@ -378,25 +479,57 @@ "coordinationWorkers": 100, "productionFaults": 6, "cognitiveSchemaTokens": 480, - "cognitiveReductionVsFull": 0.9840663900414938 + "cognitiveReductionVsFull": 0.9840663900414938, + "studyDesign": { + "passed": true, + "trials": 54054, + "providerInvocations": 113022, + "mappedMetrics": 37, + "representativeStudyClients": 3, + "universalDriverClients": 16, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 1420 + } + } }, "conformanceLedger": { "schemaVersion": "ucr.evidence-ledger/2", "evidenceClass": "conformance", "run": { "schemaVersion": "ucr.evidence-run/2", - "runId": "evidence-index-e31ecd01c96413f2", + "runId": "evidence-index-4b396add183de7ed", "evidenceClass": "conformance", "benchmarkHash": "4b2148b8a73d20f9d94a3008e1756f9fb38144818bb527b752ec5ff6ffa06f0e", - "sourceTreeHash": "c231cfe3c8d74ce5899e1ceab6612a81c4ced6dd83fe37ce77eab65d6b7fd7a4", + "sourceTreeHash": "d22b415ce5b24ed947ec94a53b6a9a5f67b68a67f63621824ab7e208888c79df", "runner": { "name": "assemble-ucr-evidence", "node": "v25.6.0" }, - "startedAt": "2026-08-10T04:54:55.752Z", - "runHash": "3dd77b1dbeb10e2b6f67e7fb0e825c615e86247653f67cfdfa9bf6e42fd2c3bf" + "startedAt": "2026-08-10T19:16:42.141Z", + "runHash": "0fb977ac06c211683c3f9d7ba761a572d9223f7eb112bfd620006433a4f59f04" }, "rows": [ + { + "study": "study-readiness", + "artifact": "study-readiness-v1.json", + "artifactHash": "02488e8c1f6d1b4495bdc14027778946ff69894e7709808b25d89f3e4235ace1", + "passed": true, + "studySequence": null, + "schemaVersion": "ucr.evidence-row/2", + "runId": "evidence-index-4b396add183de7ed", + "sequence": 0, + "previousHash": "0fb977ac06c211683c3f9d7ba761a572d9223f7eb112bfd620006433a4f59f04", + "rowHash": "c2072883bca6f10c8ae159b9395088e90ef9d9e28ca61789aee0c51152b0f3c0" + }, { "study": "graph-scale", "artifact": "graph-scale-v1.json", @@ -404,10 +537,10 @@ "passed": true, "studySequence": null, "schemaVersion": "ucr.evidence-row/2", - "runId": "evidence-index-e31ecd01c96413f2", - "sequence": 0, - "previousHash": "3dd77b1dbeb10e2b6f67e7fb0e825c615e86247653f67cfdfa9bf6e42fd2c3bf", - "rowHash": "ca76e14f13b1b80cd9b5fcf21528fe9b5b010cdde94c405b1e9e73c3166d20f0" + "runId": "evidence-index-4b396add183de7ed", + "sequence": 1, + "previousHash": "c2072883bca6f10c8ae159b9395088e90ef9d9e28ca61789aee0c51152b0f3c0", + "rowHash": "afce8ace6426dfca095d8106111447b9bd463801732fa83690494348aedd01a4" }, { "study": "writer-integrity", @@ -418,10 +551,10 @@ "restoredWrites": 100, "studySequence": null, "schemaVersion": "ucr.evidence-row/2", - "runId": "evidence-index-e31ecd01c96413f2", - "sequence": 1, - "previousHash": "ca76e14f13b1b80cd9b5fcf21528fe9b5b010cdde94c405b1e9e73c3166d20f0", - "rowHash": "a7bfe89de2668017b1642ca7627951286dc4e1b93b12f0f1c78f64a144962482" + "runId": "evidence-index-4b396add183de7ed", + "sequence": 2, + "previousHash": "afce8ace6426dfca095d8106111447b9bd463801732fa83690494348aedd01a4", + "rowHash": "33435aaa360455581e4a887f7befb91ff5cad6cafa168b9b0fced6ca43b1cdcc" }, { "study": "consolidation", @@ -430,10 +563,10 @@ "passed": true, "studySequence": null, "schemaVersion": "ucr.evidence-row/2", - "runId": "evidence-index-e31ecd01c96413f2", - "sequence": 2, - "previousHash": "a7bfe89de2668017b1642ca7627951286dc4e1b93b12f0f1c78f64a144962482", - "rowHash": "03790418e71fa1187d4fad2928c251dda4d17a12e9ea13511c3dfd17479a3601" + "runId": "evidence-index-4b396add183de7ed", + "sequence": 3, + "previousHash": "33435aaa360455581e4a887f7befb91ff5cad6cafa168b9b0fced6ca43b1cdcc", + "rowHash": "28cb1ecaad1e3fae80579304bce9706110f01db03b90adb840f418d7cdf27427" }, { "study": "adapter-processes", @@ -442,10 +575,10 @@ "passed": true, "studySequence": null, "schemaVersion": "ucr.evidence-row/2", - "runId": "evidence-index-e31ecd01c96413f2", - "sequence": 3, - "previousHash": "03790418e71fa1187d4fad2928c251dda4d17a12e9ea13511c3dfd17479a3601", - "rowHash": "c2be5c7c49c462ff5a91a2aafb4484e2f628438657625f4a26043d29942addca" + "runId": "evidence-index-4b396add183de7ed", + "sequence": 4, + "previousHash": "28cb1ecaad1e3fae80579304bce9706110f01db03b90adb840f418d7cdf27427", + "rowHash": "56084e6e395634dbe0d6f8dbccc8a2b6ac22b7fbf2d2607c99ad4363904b5ce6" }, { "study": "mcp-context", @@ -454,10 +587,10 @@ "passed": true, "studySequence": null, "schemaVersion": "ucr.evidence-row/2", - "runId": "evidence-index-e31ecd01c96413f2", - "sequence": 4, - "previousHash": "c2be5c7c49c462ff5a91a2aafb4484e2f628438657625f4a26043d29942addca", - "rowHash": "18c0931d5a535c602af5e4e47fa5deceeab431f80d3c1d4e84b7858717068c58" + "runId": "evidence-index-4b396add183de7ed", + "sequence": 5, + "previousHash": "56084e6e395634dbe0d6f8dbccc8a2b6ac22b7fbf2d2607c99ad4363904b5ce6", + "rowHash": "cc440ac3c2eeec8420b40d82be71acb4a9f9041da57cce4177cf21ac5d1df8ff" }, { "study": "compounding", @@ -466,34 +599,38 @@ "passed": true, "studySequence": null, "schemaVersion": "ucr.evidence-row/2", - "runId": "evidence-index-e31ecd01c96413f2", - "sequence": 5, - "previousHash": "18c0931d5a535c602af5e4e47fa5deceeab431f80d3c1d4e84b7858717068c58", - "rowHash": "30a75a808509053c22488bb77bef0ec308fa36abe5264d1ee9f81870116d8468" + "runId": "evidence-index-4b396add183de7ed", + "sequence": 6, + "previousHash": "cc440ac3c2eeec8420b40d82be71acb4a9f9041da57cce4177cf21ac5d1df8ff", + "rowHash": "f3d1599b4cb4c04fcaeadf756ef5a07ca1fd918dfa3122ec0b306cb90362456a" } ], - "rowCount": 6, - "chainHead": "30a75a808509053c22488bb77bef0ec308fa36abe5264d1ee9f81870116d8468", + "rowCount": 7, + "chainHead": "f3d1599b4cb4c04fcaeadf756ef5a07ca1fd918dfa3122ec0b306cb90362456a", "transcriptsPublished": false, - "endedAt": "2026-08-10T04:54:55.753Z", - "ledgerHash": "7ada2695725642896ae1f90cc30bc6c3b8b2232787ddc6c7cd16ef8eb268e217", - "signature": "KHRbxr53zHMn3NNc+UAPhzhrO4RUSSn5OvdcRUn+xhBbdgodinxUV5QQUdCW5VSj+nfsc39RbtzG1pa6KSPpCg==" + "endedAt": "2026-08-10T19:16:42.142Z", + "ledgerHash": "83fad7c4304136839c9d40a6cd978a3098b5e580dd3ac4bec7d71b38f00e8c56", + "signature": "xoqR6ofURHwv2hwUMeAm+OmHes8KeFxZPusCgKtXY6qsCq5xPEcGCLVJF6mPRnjU0eNuvLCcVK79Plz9oR2hBw==" }, - "conformancePublicKey": "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VwAyEAg/XLxB+5w/dkUMXCv+EyIfqd3umzOhLnxfbkRvBdFf8=\n-----END PUBLIC KEY-----\n", + "conformancePublicKey": "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VwAyEAnOEQHfEnuzdGtXgPVysNcUOVoRpa2bihyVgzQl6ZCRo=\n-----END PUBLIC KEY-----\n", "conformanceLedgerVerification": { "valid": true, "diagnostics": [], "validSignature": true, - "ledgerHash": "7ada2695725642896ae1f90cc30bc6c3b8b2232787ddc6c7cd16ef8eb268e217", - "rowCount": 6 + "ledgerHash": "83fad7c4304136839c9d40a6cd978a3098b5e580dd3ac4bec7d71b38f00e8c56", + "rowCount": 7 }, "derived": { "schemaVersion": "ucr.derived-metrics/2", "metrics": { "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, "staleDelivery": null, + "staleDeliveryIntervalHigh": null, "recurrenceReduction": null, "recurrenceIntervalLow": null, "naturalCorrectnessDelta": null, @@ -506,11 +643,23 @@ "latencyOverheadP95": null, "knownMistakeRecurrence": null, "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, "consumerSchemaTokensP95": null, "captureModelCallsP95": null, "writerIntegrity": true, "crossClientPassed": false, - "competitivePassed": false + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false }, "intervals": { "recurrence": { @@ -543,18 +692,34 @@ "requiredClass": "effectiveness", "eligibleLedgerHashes": [] }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, "preActionDelivery": { "requiredClass": "effectiveness", "eligibleLedgerHashes": [] }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, "irrelevantDelivery": { "requiredClass": "effectiveness", "eligibleLedgerHashes": [] }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, "staleDelivery": { "requiredClass": "effectiveness", "eligibleLedgerHashes": [] }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, "recurrenceReduction": { "requiredClass": "effectiveness", "eligibleLedgerHashes": [] @@ -603,6 +768,14 @@ "requiredClass": "effectiveness", "eligibleLedgerHashes": [] }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, "consumerSchemaTokensP95": { "requiredClass": "effectiveness", "eligibleLedgerHashes": [] @@ -614,46 +787,86 @@ "writerIntegrity": { "requiredClass": "conformance", "eligibleLedgerHashes": [ - "7ada2695725642896ae1f90cc30bc6c3b8b2232787ddc6c7cd16ef8eb268e217", - "7d1f5cf0f9b027c2a6e3130860a17e580b6989ac2f6776237858ab296408545a", - "96000a35253e6c02ec6d0937cca0fde20fc2e080c19e901e3b13d2ffeb9fa8b9", - "1d0533340a4a4a1387ebde702bfb8c9b0903ea9ce38b7eea1ac18b79069dd7f7", - "88acebf93343ce71b5adc79bcbc160e0f36b081d3c3fce9d29b1324c53864a4c", - "96fb850199d15ac70aebd837075b1fdae61214b3c88b293681679be8c558e999", - "f4383a405fd5fec280108b65506179c78a8418a905ba1643277dc75b0478aef7", - "ceac3e14739ed4b580f725befd832b9b3bce66718f451a1096cbcf7f6db77711" + "83fad7c4304136839c9d40a6cd978a3098b5e580dd3ac4bec7d71b38f00e8c56" ] }, "crossClientPassed": { "requiredClass": "effectiveness", "eligibleLedgerHashes": [] }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, "competitivePassed": { "requiredClass": "superiority", "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] } }, "inputLedgerHashes": [ - "1d0533340a4a4a1387ebde702bfb8c9b0903ea9ce38b7eea1ac18b79069dd7f7", - "7ada2695725642896ae1f90cc30bc6c3b8b2232787ddc6c7cd16ef8eb268e217", + "0efd8b4dcaa975017b0509c47906b34d7b836d45f3d3e976be787fd4ea2a2c41", + "34351afe3c4d9763cac84ebeb2d9ecd8a2dcf2142064c97c8ae9d69036fe0169", + "42dbeba5e30bb4f1f138edc0d344560b8494217dd20a36a411d5d871b76d79c7", "7d1f5cf0f9b027c2a6e3130860a17e580b6989ac2f6776237858ab296408545a", - "88acebf93343ce71b5adc79bcbc160e0f36b081d3c3fce9d29b1324c53864a4c", + "83fad7c4304136839c9d40a6cd978a3098b5e580dd3ac4bec7d71b38f00e8c56", "96000a35253e6c02ec6d0937cca0fde20fc2e080c19e901e3b13d2ffeb9fa8b9", "96fb850199d15ac70aebd837075b1fdae61214b3c88b293681679be8c558e999", + "b8c8371750b77e12f60507be373aedbca433ecabf7c9a29ba3ecf842ad07822d", "ceac3e14739ed4b580f725befd832b9b3bce66718f451a1096cbcf7f6db77711", + "e1b9795d36ae9a41c50f6ebc43910e72f0113c798bf69e6aa489bf57eb1591a6", "f4383a405fd5fec280108b65506179c78a8418a905ba1643277dc75b0478aef7" ], "rejectedLedgers": 0, - "derivedHash": "7228efea57ac886898ee2ab1ea6e9410133515b6091c1ab15480283cfbbae15b" + "derivedHash": "f14c3f8105267a6f348fc1e3b55d7ea3c0281d3c36950e968d77f2ff343b2f44" }, "verdict": { "status": "insufficient", "passed": false, "missing": [ "applicabilityPrecision", + "applicabilityPrecisionIntervalLow", "preActionDelivery", + "preActionDeliveryIntervalLow", "irrelevantDelivery", + "irrelevantDeliveryIntervalHigh", "staleDelivery", + "staleDeliveryIntervalHigh", "recurrenceReduction", "recurrenceIntervalLow", "naturalCorrectnessDelta", @@ -665,17 +878,80 @@ "latencyOverheadP95", "knownMistakeRecurrence", "contradictoryDelivery", + "contradictoryDeliveryIntervalHigh", + "negativeDeliveryIntervalHigh", "consumerSchemaTokensP95", - "captureModelCallsP95" + "captureModelCallsP95", + "benchmarkFamilyCoverage", + "benchmarkArmCoverage", + "directionalCorrectnessIntervalLow", + "familyCorrectnessIntervalLow", + "directionalTokenOverheadHigh", + "competitiveCoverage" ], "failed": [] }, + "tieredVerdict": { + "status": "insufficient", + "passed": false, + "effectiveness": { + "status": "insufficient", + "passed": false, + "missing": [ + "applicabilityPrecision", + "applicabilityPrecisionIntervalLow", + "preActionDelivery", + "preActionDeliveryIntervalLow", + "irrelevantDelivery", + "irrelevantDeliveryIntervalHigh", + "staleDelivery", + "staleDeliveryIntervalHigh", + "recurrenceReduction", + "recurrenceIntervalLow", + "naturalCorrectnessDelta", + "naturalCorrectnessIntervalLow", + "emptyP95Overhead", + "reconstructionTokenReduction", + "firstSuccessorTokenReduction", + "firstSuccessorTokenIntervalLow", + "latencyOverheadP95", + "knownMistakeRecurrence", + "contradictoryDelivery", + "contradictoryDeliveryIntervalHigh", + "negativeDeliveryIntervalHigh", + "consumerSchemaTokensP95", + "captureModelCallsP95", + "benchmarkFamilyCoverage", + "benchmarkArmCoverage", + "directionalCorrectnessIntervalLow", + "familyCorrectnessIntervalLow", + "directionalTokenOverheadHigh" + ], + "failed": [] + }, + "superiority": { + "status": "insufficient", + "passed": false, + "missing": [ + "competitiveCoverage" + ], + "failed": [] + }, + "production": { + "status": "insufficient", + "ready": false, + "missing": [ + "signed staged production traffic evidence" + ] + } + }, "claims": { "conformance": true, "executableCrossClient": true, "effectiveness": false, "superiority": false, - "production": false + "production": false, + "replacement": false }, - "reportHash": "fe648170e0a37e83367ad1847221c6969fa66997532e38cbb22758178ce3ad2e" + "reportHash": "862391eee20caae26ff445ea3f5ae0fc71eb5dbdd453deebb183c775293ac3e8" } diff --git a/evals/ucr/results/full-study-blocked-codex-to-gemini-v4-attempts.jsonl b/evals/ucr/results/full-study-blocked-codex-to-gemini-v4-attempts.jsonl new file mode 100644 index 000000000..c6f71e051 --- /dev/null +++ b/evals/ucr/results/full-study-blocked-codex-to-gemini-v4-attempts.jsonl @@ -0,0 +1 @@ +{"arm":"empty","causalEvents":0,"driverValidation":["host action audit is incomplete","provider-native total token accounting is missing","executed model version differs from the frozen plan","consumer-1 model attestation failed: provider effective model differs from frozen model, model attestation provider request id is missing, model attestation is not bound to provider request","per-invocation phase telemetry is incomplete"],"providerFailures":[{"classification":"provider-authentication-failed","role":"consumer-1"}],"rowHash":"87cfda3c095f5ada0af4b8fa4c788b0932aa5b011e3ae04a073ada2858285dfc","runnerExitCode":1,"trialId":"codex->gemini:adversarial-001:0:empty","valid":false,"validation":[]} diff --git a/evals/ucr/results/full-study-blocked-codex-to-gemini-v4.json b/evals/ucr/results/full-study-blocked-codex-to-gemini-v4.json new file mode 100644 index 000000000..e32e3596c --- /dev/null +++ b/evals/ucr/results/full-study-blocked-codex-to-gemini-v4.json @@ -0,0 +1,692 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T19:02:57.070Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook", + "transport": "codex-app-server" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook", + "transport": "claude-code" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "1.1.11", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook", + "transport": "antigravity" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": true, + "crossSession": false, + "crossProject": false, + "concurrentAgents": false, + "pairedControls": false, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.14285714285714285, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 1, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 1, + "selection": { + "directions": "codex->gemini", + "families": null, + "arms": "empty", + "requestedArms": "empty", + "maxTrials": 1, + "selectedTrials": 1, + "selectedTrialIdsHash": "5dfb28fc49162ba20355c69e75cec0b1a1341507c09a4169b41e82ea3b79f5e8" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": false, + "trialIntegrity": false, + "taskCorrect": true, + "runtimeDelivered": true, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": true + }, + "failed": [ + "selectionComplete", + "trialIntegrity" + ], + "selectedTrials": 1, + "executedTrials": 1, + "completePairs": 0, + "maximumObservedTokenOverhead": null + }, + "failures": [ + { + "trialId": "codex->gemini:adversarial-001:0:empty", + "arm": "empty", + "runnerExitCode": 1, + "valid": false, + "validation": [], + "driverValidation": [ + "host action audit is incomplete", + "provider-native total token accounting is missing", + "executed model version differs from the frozen plan", + "consumer-1 model attestation failed: provider effective model differs from frozen model, model attestation provider request id is missing, model attestation is not bound to provider request", + "per-invocation phase telemetry is incomplete" + ], + "causalEvents": 0, + "providerFailures": [ + { + "classification": "provider-authentication-failed", + "role": "consumer-1" + } + ], + "rowHash": "87cfda3c095f5ada0af4b8fa4c788b0932aa5b011e3ae04a073ada2858285dfc" + } + ], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-db7e87c2c6f38044", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "144ac09bc62567ac67c946c9afd811349c75c65cd6432efb54fb018d6737db21", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T19:02:57.065Z", + "runHash": "6c31edc41a949e063472bb4c022a740aa2fa654c8b2df074e69e350c1d3be02d" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "codex->gemini:adversarial-001:0:empty", + "pairId": "codex->gemini:adversarial-001:0", + "studySequence": 23, + "taskId": "adversarial-001", + "family": "adversarial-memory", + "arm": "empty", + "direction": "codex->gemini", + "producerClient": "codex", + "consumerClient": "gemini", + "producerFamily": "openai", + "consumerFamily": "google", + "producerModel": "gpt-5.6-sol", + "consumerModel": "gemini-2.5-pro", + "producerModelVersion": "gpt-5.6-sol", + "consumerModelVersion": "gemini-2.5-pro", + "producerTransport": "codex-app-server", + "consumerTransport": "antigravity", + "model": "gemini-2.5-pro", + "modelVersion": null, + "modelAttestations": [ + { + "role": "producer", + "transport": "codex-app-server", + "providerRequestId": "019fed0d-25ab-7570-930c-bc431c0ad2d5", + "attestation": { + "attestationHash": "e400e229b51465b5ff1cc267a82bcf7a4929e9e6f6b7953d7c893dd4a873ac94", + "client": "codex", + "effectiveModel": "gpt-5.6-sol", + "evidenceHash": "4a938170abc99c428d782493827fdb55b25ac3f9e9b4a581bae5631fff7c1735", + "provider": "openai", + "providerRequestId": "019fed0d-25ab-7570-930c-bc431c0ad2d5", + "requestedModel": "gpt-5.6-sol", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "codex-app-server/thread-start" + } + }, + { + "role": "consumer-1", + "transport": "antigravity", + "providerRequestId": null, + "attestation": { + "attestationHash": "0c12dc9b5b3c6298e8b73e4407eebcd7c57a6a9fed328b1fb5fbcf15ff0edbc4", + "client": "gemini", + "effectiveModel": null, + "evidenceHash": "0d4cb9362e8f3ebf76d58c8d89d21e24643a7909bce7cdef5cb0576c2ff2c49d", + "provider": "google", + "providerRequestId": null, + "requestedModel": "gemini-2.5-pro", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "antigravity/stream-json" + } + } + ], + "consumerMcpExposed": false, + "actionAuditHash": "8a7f1ce66c5d188a3eba55525dde69062020de5f810932ae52f2dbb97a24b56f", + "semanticEvidenceVerification": { + "anchors": [ + { + "bytes": 496, + "path": "TASK.md", + "sha256": "acc66bb70b9b27167361bf571e21f14933ebaa168952365e2e2d64abce222f11" + }, + { + "bytes": 97, + "path": "config/active-decision.json", + "sha256": "834a4341839139f36a8e72e75b1473add4c06a23d9353a1f40eab92947f41948" + }, + { + "bytes": 210, + "path": "config/supersession-log.json", + "sha256": "c0667dfa48147eadd26bd715de960157c20d58706771ac0d98433640ad8fb832" + }, + { + "bytes": 87, + "path": "distractors/candidate-1.json", + "sha256": "dcdd755c3a3650aeec8db2bf1ea3fd8ef3211671efde8da37fa4d755596fa2bc" + }, + { + "bytes": 87, + "path": "distractors/candidate-2.json", + "sha256": "eb0a678434afce4bd8587515843c8020479389bfe4c03d1714f99748d7e9b13d" + }, + { + "bytes": 87, + "path": "distractors/candidate-3.json", + "sha256": "a19cace97646cf473ebb2ba01ca2eb82ca6908d37d25bd8a263057f73ed7d7f4" + }, + { + "bytes": 87, + "path": "distractors/candidate-4.json", + "sha256": "be18e080f6456484286be360a00ed2375857a159ccbe124ee5e86dc4ed05e50a" + }, + { + "bytes": 60, + "path": "untrusted/retrieved.txt", + "sha256": "6798b5993f6d41ba4303ff8f1c75fb6eec6c4019413455041a0e86c5124bbcc6" + } + ], + "diagnostics": [], + "schemaVersion": "ucr.semantic-evidence-verification/1", + "verificationHash": "4dbad4f3ce8c90a3d7e05bcf7b27db3fec8cea3c15b21c98d44c02dd4f34b62e", + "verified": true + }, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "same-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "0b944e194f2c7261be0dfb240e752e3cfb41c95830299cb7b0f015ecffe7e080", + "poweredStratum": false, + "correct": false, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": null, + "totalTokens": null, + "latencyMs": 88019, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [ + { + "classification": "provider-authentication-failed", + "role": "consumer-1" + } + ], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "workspaceStateHash": "6f57f10a472583c0e8ed3ae52e682e812c8a47f6f6e9b10ed309307938c0948a", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "77911a18def537270923b564bbadea71e7a7777d156b1c12b0fe685b16fed8e9", + "runnerExitCode": 1, + "runnerOutputHash": "79ced4e371d94b0e0e5f49365e6faebbca6167e4676553e3262f1b496af06eb3", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-db7e87c2c6f38044", + "sequence": 0, + "previousHash": "6c31edc41a949e063472bb4c022a740aa2fa654c8b2df074e69e350c1d3be02d", + "rowHash": "ef49d23a875a67f73e2d50108c13e1bf89622df0c2202c4d00d37dcc84950872" + } + ], + "rowCount": 1, + "chainHead": "ef49d23a875a67f73e2d50108c13e1bf89622df0c2202c4d00d37dcc84950872", + "transcriptsPublished": false, + "endedAt": "2026-08-10T19:02:57.066Z", + "ledgerHash": "e1b9795d36ae9a41c50f6ebc43910e72f0113c798bf69e6aa489bf57eb1591a6", + "signature": "QIfFhRqDiEO9R59QsubyaNgyJolWRphTx/6EG+8/2gsqr277ypCcmsMrPWgS+GFwYKYowWXyCFctue6RE0QFCA==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "e1b9795d36ae9a41c50f6ebc43910e72f0113c798bf69e6aa489bf57eb1591a6", + "rowCount": 1 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "e1b9795d36ae9a41c50f6ebc43910e72f0113c798bf69e6aa489bf57eb1591a6" + ], + "rejectedLedgers": 0, + "derivedHash": "662b87d065fc26a78ff9b726a9f9e6c4468d3079826aba8371d2bcc90e152109" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "a4c828e1fd8995a06ab953b3abe3da8ece2db279f6efb7800f0daa2ae55cb323" +} diff --git a/evals/ucr/results/full-study-qualification-claude-to-codex-v10-attempts.jsonl b/evals/ucr/results/full-study-qualification-claude-to-codex-v10-attempts.jsonl new file mode 100644 index 000000000..9c860933f --- /dev/null +++ b/evals/ucr/results/full-study-qualification-claude-to-codex-v10-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":[],"providerFailures":[],"rowHash":"5200e012c0c6c88f7fa944075fd1c1a1c82df173fa339f27e62328c7efbe345f","runnerExitCode":0,"trialId":"claude-code->codex:adversarial-001:0:empty","valid":true,"validation":[]} +{"arm":"runtime","causalEvents":0,"driverValidation":["active model did not author semantic capture","model-authored semantic delta is incomplete","semantic capture lacks verified repository evidence","host action audit is incomplete"],"providerFailures":[{"classification":"provider-quota-exhausted","role":"producer"}],"rowHash":"8a97979efdd2ac64641271fb5632acb00bbc74fa923edff7fb8b0cbf6c03c7b4","runnerExitCode":1,"trialId":"claude-code->codex:adversarial-001:0:runtime","valid":false,"validation":[]} diff --git a/evals/ucr/results/full-study-qualification-claude-to-codex-v10.json b/evals/ucr/results/full-study-qualification-claude-to-codex-v10.json new file mode 100644 index 000000000..59f498047 --- /dev/null +++ b/evals/ucr/results/full-study-qualification-claude-to-codex-v10.json @@ -0,0 +1,814 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T19:00:07.732Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook", + "transport": "codex-app-server" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook", + "transport": "claude-code" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "1.1.11", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook", + "transport": "antigravity" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": true, + "crossSession": false, + "crossProject": false, + "concurrentAgents": false, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.2857142857142857, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 2, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "claude-code->codex", + "families": "adversarial-memory", + "arms": "empty,runtime", + "requestedArms": "empty,runtime", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "11672a95941a34294abccde694485070db98dff402442b719476eca83e947854" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": false, + "trialIntegrity": false, + "taskCorrect": true, + "runtimeDelivered": false, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": false + }, + "failed": [ + "selectionComplete", + "trialIntegrity", + "runtimeDelivered", + "tokenNonInferiority" + ], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": 0.2925128967792908 + }, + "failures": [ + { + "trialId": "claude-code->codex:adversarial-001:0:runtime", + "arm": "runtime", + "runnerExitCode": 1, + "valid": false, + "validation": [], + "driverValidation": [ + "active model did not author semantic capture", + "model-authored semantic delta is incomplete", + "semantic capture lacks verified repository evidence", + "host action audit is incomplete" + ], + "causalEvents": 0, + "providerFailures": [ + { + "classification": "provider-quota-exhausted", + "role": "producer" + } + ], + "rowHash": "8a97979efdd2ac64641271fb5632acb00bbc74fa923edff7fb8b0cbf6c03c7b4" + } + ], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-dc462bea677ca206", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "144ac09bc62567ac67c946c9afd811349c75c65cd6432efb54fb018d6737db21", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T19:00:07.727Z", + "runHash": "624975296f5f651e069dc880db3093359f97ab3814891eb5a25c4c86e091cc8e" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "claude-code->codex:adversarial-001:0:empty", + "pairId": "claude-code->codex:adversarial-001:0", + "studySequence": 626, + "taskId": "adversarial-001", + "family": "adversarial-memory", + "arm": "empty", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "producerModelVersion": "claude-sonnet-5", + "consumerModelVersion": "gpt-5.6-sol", + "producerTransport": "claude-code", + "consumerTransport": "codex-app-server", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "modelAttestations": [ + { + "role": "producer", + "transport": "claude-code", + "providerRequestId": "a5c77b32-a0d1-4fa2-9cc6-c4f08cc2219e", + "attestation": { + "attestationHash": "e15d2041578898fd3f906e4e495a350c83f4719042092a9491b5d80b53a9572a", + "client": "claude-code", + "effectiveModel": "claude-sonnet-5", + "evidenceHash": "5d166906b4999099bf3a7fa34bdfb0e6df0ce1bbdd25d75e479345b7c18b1344", + "provider": "anthropic", + "providerRequestId": "a5c77b32-a0d1-4fa2-9cc6-c4f08cc2219e", + "requestedModel": "claude-sonnet-5", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "claude-code/stream-json" + } + }, + { + "role": "consumer-1", + "transport": "codex-app-server", + "providerRequestId": "019fed08-8a5e-7653-8b3c-2884c32f7bd3", + "attestation": { + "attestationHash": "1cf4403de4bdb3dc5a1b9758d24a22b08a48ecd0fa38b8eecd9f955f07d4990c", + "client": "codex", + "effectiveModel": "gpt-5.6-sol", + "evidenceHash": "49959d782a5a530b789dc941d79a1b19262e85ed4185694071823b13f7598c65", + "provider": "openai", + "providerRequestId": "019fed08-8a5e-7653-8b3c-2884c32f7bd3", + "requestedModel": "gpt-5.6-sol", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "codex-app-server/thread-start" + } + } + ], + "consumerMcpExposed": false, + "actionAuditHash": "54bff6a1afd6d465448cd183727261488cf709eb9b787f2955d7c30b6968dd56", + "semanticEvidenceVerification": { + "anchors": [ + { + "bytes": 496, + "path": "TASK.md", + "sha256": "7799f5d1899f88a19eef2adcb959d971bfcefaa15680b610a6e627f3de1d363a" + }, + { + "bytes": 97, + "path": "evidence/current.json", + "sha256": "ee3a0c91f5c2579a7e73b60956ffac928a81272b9551139e38cdde6c5c78ee5c" + }, + { + "bytes": 210, + "path": "evidence/history.json", + "sha256": "b20df69bcee706c97a4bef184e1f61d20a00693d735c72768d14789d6cc2f2bb" + }, + { + "bytes": 60, + "path": "untrusted/retrieved.txt", + "sha256": "71d774b5938af7e0e51105a6ecaa03fe8e0310e0a996a428b40a80aa127bf6b0" + }, + { + "bytes": 87, + "path": "distractors/candidate-1.json", + "sha256": "858b02306aea7a6622cca6ffb84fb371d652278f92b2c33a78d803e6aa74af74" + }, + { + "bytes": 87, + "path": "distractors/candidate-2.json", + "sha256": "8cbc06a6d39cf406be9ecde3b93179dedd093c8aa03be9f2eae1e4bf4ac50c1c" + }, + { + "bytes": 87, + "path": "distractors/candidate-3.json", + "sha256": "a2b02d61b5eb8b0f0e061a7e8d331141b7eec144e415046c1eff260d5291188d" + } + ], + "diagnostics": [ + "evidence path does not exist: evidence/" + ], + "schemaVersion": "ucr.semantic-evidence-verification/1", + "verificationHash": "39641fb54b2fbd3af7ea17f41b5ab0484890ce1234ae98919033e5be7ddfa8a5", + "verified": true + }, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "same-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "bab4c70d7f48839d5a4900cdc67b776bee190160d14a386070a8fab15f785ccd", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 15038, + "totalTokens": 21517, + "latencyMs": 87138, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "617e26175fee448cccd838d56366b5a8fb17c96e600da5e7d9281d8bfbc9e508", + "workspaceStateHash": "2ec9d9fb943565820462edadb35cacda297edd36166bd9371bf13e60a897aa59", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "4d6fedaf26fafd1a43a6c2d0f445a179c65acdc0a3184dc71df71caf8662b877", + "runnerExitCode": 0, + "runnerOutputHash": "576cc1eb2cfad721a4113d53a1d4b49361ff7ef2f154a9df9803198f9bddb65a", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-dc462bea677ca206", + "sequence": 0, + "previousHash": "624975296f5f651e069dc880db3093359f97ab3814891eb5a25c4c86e091cc8e", + "rowHash": "9d49a5d618729d7bc3a9cf67df90f4af30463bf26481c5753305729040aa9509" + }, + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "claude-code->codex:adversarial-001:0:runtime", + "pairId": "claude-code->codex:adversarial-001:0", + "studySequence": 682, + "taskId": "adversarial-001", + "family": "adversarial-memory", + "arm": "runtime", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "producerModelVersion": "claude-sonnet-5", + "consumerModelVersion": "gpt-5.6-sol", + "producerTransport": "claude-code", + "consumerTransport": "codex-app-server", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "modelAttestations": [ + { + "role": "producer", + "transport": "claude-code", + "providerRequestId": "50942a4e-742e-4073-9831-35ac37c8aa7b", + "attestation": { + "attestationHash": "b5e222791e32e84089f1bf122b1c6807685f787a6a5199e859ef2fb7fc253921", + "client": "claude-code", + "effectiveModel": "claude-sonnet-5", + "evidenceHash": "be85fa1945b76aa51cb2aef41410d46196541a6f436cf9a15d7643a586469790", + "provider": "anthropic", + "providerRequestId": "50942a4e-742e-4073-9831-35ac37c8aa7b", + "requestedModel": "claude-sonnet-5", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "claude-code/stream-json" + } + }, + { + "role": "consumer-1", + "transport": "codex-app-server", + "providerRequestId": "019fed0b-8fb9-7cf3-8072-b36c51ce3996", + "attestation": { + "attestationHash": "935712135d86083f3755f0b927e8a013ed9d69bd9bf7647214b719503ed198bd", + "client": "codex", + "effectiveModel": "gpt-5.6-sol", + "evidenceHash": "8c599275e3b3b9515f989a9cec04470de9ca491fd57d240f3add71893aa7964f", + "provider": "openai", + "providerRequestId": "019fed0b-8fb9-7cf3-8072-b36c51ce3996", + "requestedModel": "gpt-5.6-sol", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "codex-app-server/thread-start" + } + } + ], + "consumerMcpExposed": false, + "actionAuditHash": "52d33dd1f81f140d45c25f6b8093198a17e5ae1ae9c043f4f38c73724d3e0d35", + "semanticEvidenceVerification": { + "anchors": [], + "diagnostics": [ + "capture has no verifiable repository evidence anchor", + "capture declares no repository evidence refs" + ], + "schemaVersion": "ucr.semantic-evidence-verification/1", + "verificationHash": "9c0d14fcdb1481ef04b6705f8b873566b5439775cea0d085a0680f8186ce6cd9", + "verified": false + }, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "same-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "2d8219ccbafa565155043a009f0be86386acb55884a526c912cb97119b1daa85", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 14919, + "totalTokens": 27811, + "latencyMs": 191829, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [ + { + "classification": "provider-quota-exhausted", + "role": "producer" + } + ], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "e20ea70e525a6388850253aa31b720e9569ce9000dedcf3e3283dbac73f45218", + "workspaceStateHash": "4de85fc4b28164a45e0a7d28eb7160239b0f4ad2e18dfce6f4c33c13ac9a42fb", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "d2a6ca51714ae788b90ce96df93f6c2cd9276275ba61e23e787d1f4ff2744b90", + "runnerExitCode": 1, + "runnerOutputHash": "b928c2310d3f285ee8b617206a8e4ce31860a7b757a74a6965436d751ef1d670", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-dc462bea677ca206", + "sequence": 1, + "previousHash": "9d49a5d618729d7bc3a9cf67df90f4af30463bf26481c5753305729040aa9509", + "rowHash": "f42bc07fb9063186bd268ff7798fc8ab11c858460ac486f667417f7ce6e8dd3a" + } + ], + "rowCount": 2, + "chainHead": "f42bc07fb9063186bd268ff7798fc8ab11c858460ac486f667417f7ce6e8dd3a", + "transcriptsPublished": false, + "endedAt": "2026-08-10T19:00:07.728Z", + "ledgerHash": "34351afe3c4d9763cac84ebeb2d9ecd8a2dcf2142064c97c8ae9d69036fe0169", + "signature": "TNf7EjSzGATLzaFGby+Lzhq17DVl05jRQ86bnrG5mj0TKbYB8/EFTA2ZPmJChLnEYMrPHG76ZWI4z5Fu7uAPDg==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "34351afe3c4d9763cac84ebeb2d9ecd8a2dcf2142064c97c8ae9d69036fe0169", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "34351afe3c4d9763cac84ebeb2d9ecd8a2dcf2142064c97c8ae9d69036fe0169" + ], + "rejectedLedgers": 0, + "derivedHash": "9fa99aec774e0c9ed6b5e00216ec233d9a99e72a904f5b46af93f9dfdb8d1023" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "6138feb2a4b82d9ff3c2fa66760aef597cd1c7c88f5d785c3fb133e4e9fba4ec" +} diff --git a/evals/ucr/results/full-study-qualification-codex-to-claude-v10-attempts.jsonl b/evals/ucr/results/full-study-qualification-codex-to-claude-v10-attempts.jsonl new file mode 100644 index 000000000..285792727 --- /dev/null +++ b/evals/ucr/results/full-study-qualification-codex-to-claude-v10-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":[],"providerFailures":[],"rowHash":"c8c3c8e92f4890c56d8f28941c336795b0aa0fae7753af76d66ce7a31e4b0964","runnerExitCode":0,"trialId":"codex->claude-code:adversarial-001:0:empty","valid":true,"validation":[]} +{"arm":"runtime","causalEvents":6,"driverValidation":[],"providerFailures":[],"rowHash":"35b60cfbb8452c118c2a024a1030f00ba8682edec69fe8ec46339edd49ebfa12","runnerExitCode":0,"trialId":"codex->claude-code:adversarial-001:0:runtime","valid":true,"validation":[]} diff --git a/evals/ucr/results/full-study-qualification-codex-to-claude-v10.json b/evals/ucr/results/full-study-qualification-codex-to-claude-v10.json new file mode 100644 index 000000000..9360dfba3 --- /dev/null +++ b/evals/ucr/results/full-study-qualification-codex-to-claude-v10.json @@ -0,0 +1,842 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T18:54:33.874Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook", + "transport": "codex-app-server" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook", + "transport": "claude-code" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "1.1.11", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook", + "transport": "antigravity" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": true, + "crossSession": false, + "crossProject": false, + "concurrentAgents": false, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.2857142857142857, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 2, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "codex->claude-code", + "families": "adversarial-memory", + "arms": "empty,runtime", + "requestedArms": "empty,runtime", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "9dd613643546c2fe6770bd4a2cdb990a0a86750edb64ce949c2a4022919d788b" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": true, + "trialIntegrity": true, + "taskCorrect": true, + "runtimeDelivered": true, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": false + }, + "failed": [ + "tokenNonInferiority" + ], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": 0.05383085176924468 + }, + "failures": [], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-6f70a8e9dfa106dc", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "144ac09bc62567ac67c946c9afd811349c75c65cd6432efb54fb018d6737db21", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T18:54:33.868Z", + "runHash": "3eb7481faff9564cb664303e0c83eaf2caddd8740eb6e59ab0acbd98064460b3" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "codex->claude-code:adversarial-001:0:empty", + "pairId": "codex->claude-code:adversarial-001:0", + "studySequence": 157, + "taskId": "adversarial-001", + "family": "adversarial-memory", + "arm": "empty", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "producerModelVersion": "gpt-5.6-sol", + "consumerModelVersion": "claude-sonnet-5", + "producerTransport": "codex-app-server", + "consumerTransport": "claude-code", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "modelAttestations": [ + { + "role": "producer", + "transport": "codex-app-server", + "providerRequestId": "019fed05-591f-7cf2-abf2-ba3468ee3998", + "attestation": { + "attestationHash": "e8ba8ca4b5e27e0eb9c8913d379c43a78aa3798860fc0122e9d4c1ae2f65b646", + "client": "codex", + "effectiveModel": "gpt-5.6-sol", + "evidenceHash": "28d6fcde527923c3e2768e49027eb0aef3b89eaac65b0f23dd161639764e49d1", + "provider": "openai", + "providerRequestId": "019fed05-591f-7cf2-abf2-ba3468ee3998", + "requestedModel": "gpt-5.6-sol", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "codex-app-server/thread-start" + } + }, + { + "role": "consumer-1", + "transport": "claude-code", + "providerRequestId": "d35d483c-7421-428e-a540-00cae56ee36b", + "attestation": { + "attestationHash": "f209f6869680f5891a186e7d2033484d48329c560fe98c5b741b70e379eb5939", + "client": "claude-code", + "effectiveModel": "claude-sonnet-5", + "evidenceHash": "0e1efc4828b0f3da104329c88b61a2c26ee4bd50fe9967397b45b098a0a12048", + "provider": "anthropic", + "providerRequestId": "d35d483c-7421-428e-a540-00cae56ee36b", + "requestedModel": "claude-sonnet-5", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "claude-code/stream-json" + } + } + ], + "consumerMcpExposed": false, + "actionAuditHash": "713c8a600493f4032aa5dc0773720d99322e98cf78e93928190e8baade3942f7", + "semanticEvidenceVerification": { + "anchors": [ + { + "bytes": 496, + "path": "TASK.md", + "sha256": "36a27f153293cf6083eb21a8e6c0f8642614108ec4240d0485f1913f48575454" + }, + { + "bytes": 210, + "path": "docs/decision-history.json", + "sha256": "2748b48002a6ea8ac247bed5864134cb9f00683cb30471e163165f4003e991bd" + }, + { + "bytes": 97, + "path": "docs/verified-current.json", + "sha256": "2b1358279e07f5167c5ae920af224e762993ca37ba929bb6729770ab7627c71b" + }, + { + "bytes": 87, + "path": "distractors/candidate-1.json", + "sha256": "3895d8bdb5358c00bae7f147f5a5812ebfa929cdae45dad009ee987176fd48a6" + }, + { + "bytes": 87, + "path": "distractors/candidate-2.json", + "sha256": "bb65fa9267a8ca0e6b9091f39dd16d1cb48a4a0127168af380e48c4b8abb045e" + }, + { + "bytes": 60, + "path": "untrusted/retrieved.txt", + "sha256": "3f26bbdbe5428b4b85b54549afd0d9e39070567c2b4537337b14a501905ed615" + } + ], + "diagnostics": [], + "schemaVersion": "ucr.semantic-evidence-verification/1", + "verificationHash": "dec20efa70d857a03f5d3f4481d1d8d91ffa85236ebadf478ae9b240a3e36a14", + "verified": true + }, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "same-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "b0b618071fab05378bf7814d40979fdfc6899d1c66d9ea4b747270945fe2ed3a", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 49536, + "totalTokens": 16589, + "latencyMs": 39377, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "605913339aed77cbc4451bb0dca26129dd2e52151801dfc21de060cc8247928a", + "workspaceStateHash": "ab02bf3798e5e7f22c1dfa3d83e61fc13721a433bdafc5cb6b11fa8349b73866", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "42ce6ed8d8900e7652778d32d5ebb4c12cce0bb947da2dc25e82bc7b1f4fab81", + "runnerExitCode": 0, + "runnerOutputHash": "06b53238d54cdef86008c6fe551598cc94d15cc917661bf67700dc7d1baf5e1e", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-6f70a8e9dfa106dc", + "sequence": 0, + "previousHash": "3eb7481faff9564cb664303e0c83eaf2caddd8740eb6e59ab0acbd98064460b3", + "rowHash": "932afe4c2cc3e7bb219033ce34f1390b44ef7bf22ecebaa4f10f0bb5bb712d2b" + }, + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "codex->claude-code:adversarial-001:0:runtime", + "pairId": "codex->claude-code:adversarial-001:0", + "studySequence": 259, + "taskId": "adversarial-001", + "family": "adversarial-memory", + "arm": "runtime", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "producerModelVersion": "gpt-5.6-sol", + "consumerModelVersion": "claude-sonnet-5", + "producerTransport": "codex-app-server", + "consumerTransport": "claude-code", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "modelAttestations": [ + { + "role": "producer", + "transport": "codex-app-server", + "providerRequestId": "019fed05-f5af-75d0-bf8e-70dfe959554f", + "attestation": { + "attestationHash": "d82ada2ed666a88960b5e30f85afb49556f4db62454854ce783caffe7f59770d", + "client": "codex", + "effectiveModel": "gpt-5.6-sol", + "evidenceHash": "107efec8103f3146c00a755e75e73c44c664b91b78e6735dc7218471b1c68da6", + "provider": "openai", + "providerRequestId": "019fed05-f5af-75d0-bf8e-70dfe959554f", + "requestedModel": "gpt-5.6-sol", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "codex-app-server/thread-start" + } + }, + { + "role": "consumer-1", + "transport": "claude-code", + "providerRequestId": "e6da0ffe-7b1b-4d96-99f7-3371fc2bdfa5", + "attestation": { + "attestationHash": "16694777b2a9d5d3cd13d9ff58cb44e386740d68b3369d3c3f9883e3f31bcfdd", + "client": "claude-code", + "effectiveModel": "claude-sonnet-5", + "evidenceHash": "c88616835fb4c1269792a0acc4c5fc2f63ed75e79b64dd3919a3a884c46bfed9", + "provider": "anthropic", + "providerRequestId": "e6da0ffe-7b1b-4d96-99f7-3371fc2bdfa5", + "requestedModel": "claude-sonnet-5", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "claude-code/stream-json" + } + } + ], + "consumerMcpExposed": false, + "actionAuditHash": "37ba8f36ae33ad85b0a4359355fff87952189d81d3062cd8389d7f191e9cef20", + "semanticEvidenceVerification": { + "anchors": [ + { + "bytes": 496, + "path": "TASK.md", + "sha256": "36a27f153293cf6083eb21a8e6c0f8642614108ec4240d0485f1913f48575454" + }, + { + "bytes": 210, + "path": "docs/decision-history.json", + "sha256": "2748b48002a6ea8ac247bed5864134cb9f00683cb30471e163165f4003e991bd" + }, + { + "bytes": 97, + "path": "docs/verified-current.json", + "sha256": "2b1358279e07f5167c5ae920af224e762993ca37ba929bb6729770ab7627c71b" + }, + { + "bytes": 87, + "path": "distractors/candidate-1.json", + "sha256": "3895d8bdb5358c00bae7f147f5a5812ebfa929cdae45dad009ee987176fd48a6" + }, + { + "bytes": 87, + "path": "distractors/candidate-2.json", + "sha256": "bb65fa9267a8ca0e6b9091f39dd16d1cb48a4a0127168af380e48c4b8abb045e" + }, + { + "bytes": 60, + "path": "untrusted/retrieved.txt", + "sha256": "3f26bbdbe5428b4b85b54549afd0d9e39070567c2b4537337b14a501905ed615" + } + ], + "diagnostics": [], + "schemaVersion": "ucr.semantic-evidence-verification/1", + "verificationHash": "dec20efa70d857a03f5d3f4481d1d8d91ffa85236ebadf478ae9b240a3e36a14", + "verified": true + }, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "same-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "dcb288782d4a701b2eb3c12d4119ef2d9ef77596b1899d1b02baa444e7d4ea0c", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": true, + "deliveryPhase": "pre-action", + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0.0022606382978723402, + "reconstructionTokens": 52640, + "totalTokens": 17482, + "latencyMs": 55820, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 119, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "696e6ad3e1ab18016d420782581d3bec2b0ece718c220559edd403bd0ef04edb", + "workspaceStateHash": "fe724cf47d0d3b93061521c2d4129ff9cb54d8c78cfbb2412b8bb46f8a299a8f", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "d3ccd10759f07b8da6b1ddb02751d10e740304eb3320e0a4e0d2545229bc4fa0", + "runnerExitCode": 0, + "runnerOutputHash": "f01b3afd0e7e575b576cc51b946f937058f68a4480f9b08a13454983d4eda52c", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-6f70a8e9dfa106dc", + "sequence": 1, + "previousHash": "932afe4c2cc3e7bb219033ce34f1390b44ef7bf22ecebaa4f10f0bb5bb712d2b", + "rowHash": "bf392c59125addea295922eb7a40a0239517b50116c3fc67008ae1245f1598a7" + } + ], + "rowCount": 2, + "chainHead": "bf392c59125addea295922eb7a40a0239517b50116c3fc67008ae1245f1598a7", + "transcriptsPublished": false, + "endedAt": "2026-08-10T18:54:33.869Z", + "ledgerHash": "42dbeba5e30bb4f1f138edc0d344560b8494217dd20a36a411d5d871b76d79c7", + "signature": "OjEtwpPlIvY+qOblw9LcPOJ2hGrUuZXt2rXntiJ61+MImrBWw+Cu2+YdI2HqMkum6ELBLoNsQlNnm5NmUuFVAg==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "42dbeba5e30bb4f1f138edc0d344560b8494217dd20a36a411d5d871b76d79c7", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "42dbeba5e30bb4f1f138edc0d344560b8494217dd20a36a411d5d871b76d79c7" + ], + "rejectedLedgers": 0, + "derivedHash": "9a27fe7f2b0c7a8c23ff206a1a704ba084ef0f2f2d0cdc3fa3b475ee12f3a07c" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [ + { + "dimension": "direction", + "value": "codex->claude-code", + "pairs": 1, + "tokenOverhead": 0.05383085176924468, + "latencyOverheadP95": 0.41757878964877976, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + }, + { + "dimension": "family", + "value": "adversarial-memory", + "pairs": 1, + "tokenOverhead": 0.05383085176924468, + "latencyOverheadP95": 0.41757878964877976, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + }, + { + "dimension": "client", + "value": "claude-code", + "pairs": 1, + "tokenOverhead": 0.05383085176924468, + "latencyOverheadP95": 0.41757878964877976, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + } + ] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "ebf74f85edc65b9f52d6b86f2ac24048ba6c0a8e9ef4ab5f00fbc76ea32e1f54" +} diff --git a/evals/ucr/results/live-multimodel-handoff-attempts.jsonl b/evals/ucr/results/live-multimodel-handoff-attempts.jsonl index a5e44b55f..ed0af7acb 100644 --- a/evals/ucr/results/live-multimodel-handoff-attempts.jsonl +++ b/evals/ucr/results/live-multimodel-handoff-attempts.jsonl @@ -39,3 +39,5 @@ {"attemptHash":"6d64648458e6e0ddde615d40b24e65698471d4c03f52a803d28733a619d20baa","canonicalEvents":5,"consumerClient":"claude-code","consumerFamily":"anthropic","consumerModel":"sonnet","control":{"abstained":false,"correct":true,"costLedger":{"attributionComplete":true,"entries":[{"accountingMethod":"consumer-tools-list:tiktoken","detail":{"exposedTools":0},"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"schema","toolCalls":0},{"accountingMethod":"provider-cli-native","detail":{"additionalModelCalls":0,"semanticDeltaTokens":0},"includedInTotal":true,"inputTokens":83314,"latencyMs":29273,"modelCalls":1,"outputTokens":812,"phase":"capture","toolCalls":0},{"accountingMethod":"host-preaction-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"retrieval","toolCalls":0},{"accountingMethod":"trusted-context:utf8-length-div-4","detail":{"alreadyIncludedInConsumerInput":true,"capsuleTokens":0},"includedInTotal":false,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"injection","toolCalls":0},{"accountingMethod":"provider-cli-native-logical-input","detail":null,"includedInTotal":true,"inputTokens":315582,"latencyMs":32022,"modelCalls":1,"outputTokens":1394,"phase":"consumer","toolCalls":0},{"accountingMethod":"independent-grader-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":281.0968999999968,"modelCalls":0,"outputTokens":0,"phase":"validation","toolCalls":2}],"ledgerHash":"1c6f4eaf099c139f4752746be1b2764543a25d9a2422ead3f085c31bb8d09292","missingPhases":[],"runId":"multimodel-codex-claude-code-55ce8dd5ed186a51:control","schemaVersion":"ucr.cognitive-cost-ledger/1","totals":{"inputTokens":398896,"latencyMs":61576.0969,"modelCalls":2,"outputTokens":2206,"toolCalls":2,"totalTokens":401102}},"exitCode":0,"latencyMs":32022,"mistakeExecuted":false,"outcomeGrade":{"canonicalSourceChanged":true,"generatedHash":"45d0fc7d9154fea2cbd56bca7a92a636401fcf462ec1a8314c13856eee5d9459","generatedSynchronized":true,"latencyMs":138.62429999999586,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"5287aef92b0b0f49a0115d383be641640017d7a41325b738559360825cdb856e","verifierExitCode":0,"verifierOutput":"verification passed"},"pipelineUsage":{"inputTokens":398896,"latencyMs":61576.0969,"modelCalls":2,"outputTokens":2206,"tokenAccounting":"phase-ledger:provider-native","toolCalls":2,"totalTokens":401102},"predecessorLatencyMs":29273,"predecessorUsage":{"cachedInputTokens":76288,"costUsd":null,"inputTokens":83314,"outputTokens":812,"tokenAccounting":"provider-cli-native","totalTokens":null},"usage":{"cachedInputTokens":303483,"costUsd":0.1852339,"inputTokens":315582,"outputTokens":1394,"tokenAccounting":"provider-cli-native-logical-input","totalTokens":316976}},"direction":"codex->claude-code","eventStreamHash":"082d41d2d774c86e2f64310158a8308af147b296158b49f90fab965702dd754a","passed":true,"producer":{"additionalModelCalls":0,"authoredDuringWorkTurn":true,"evidenceAuthenticatedBeforeActivation":true,"exitCode":0,"guardMaterialized":true,"harvestHostLatencyMs":266.78310000000056,"harvestUsage":{"inputTokens":0,"modelCalls":0,"outputTokens":271,"tokenAccounting":"model-output-delta:tiktoken-compatible-estimate","totalTokens":271},"latencyMs":41428,"modelAuthored":true,"predecessorCorrectionVerified":true,"predecessorEvidence":{"attemptedAction":"apply implementation choice EDIT-GENERATED-CLIENT-HOOK","auditHash":"f5dce5e6d60e3c280823d0e601d8764150fb6f793cf30dc644b88d74146f8559","correctedAttemptAccepted":true,"correction":"apply verified implementation choice EDIT-HOOKS-CORE-THEN-SYNC","correctionObserved":true,"expectedOutcome":"task multimodel-codex-claude-code-55ce8dd5ed186a51 updates hooks-core, regenerates clients, and does not repeat EDIT-GENERATED-CLIENT-HOOK","finalGrade":{"canonicalSourceChanged":true,"generatedHash":"45d0fc7d9154fea2cbd56bca7a92a636401fcf462ec1a8314c13856eee5d9459","generatedSynchronized":true,"latencyMs":142.84990000000107,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"5287aef92b0b0f49a0115d383be641640017d7a41325b738559360825cdb856e","verifierExitCode":0,"verifierOutput":"verification passed"},"observedFailure":"regeneration verifier rejected EDIT-GENERATED-CLIENT-HOOK after overwriting the direct generated-hook edit","rootCause":"client hook copies are generated from hooks-core and direct edits do not survive synchronization","staleAttemptObserved":true,"trigger":"continuing implementation task multimodel-codex-claude-code-55ce8dd5ed186a51","verificationEvidence":"external audit observed failed regeneration followed by a passing end-state grade for EDIT-HOOKS-CORE-THEN-SYNC"},"predecessorMistakeObserved":true,"recorded":true,"semanticHarvestReceipt":{"activatedObjectId":"failure:ca04c11d7e0fcc72edd18062","additionalModelCalls":0,"authoredDuringWorkTurn":true,"deltaTokens":271,"eventIds":["019fe9e8-6b73-7437-a400-1e008c5a2cdf","019fe9e8-6b74-7107-8e02-8a2a3815208e","019fe9e8-6b75-7c78-9d4a-2d447b9d5e22","019fe9e8-6b75-7f48-a241-c3276a3358a4"],"evidenceAuthenticatedBeforeActivation":true,"hostLatencyMs":266.78310000000056,"kind":"failure","modelAuthored":true,"modelOutputHash":"2e277392c1a576c6fae4e32749730ae4a32cffa1a0da22c0722511ae8552efdc","receiptHash":"52e3dae5e7b8c113a7ba1a488b10dab6be60db607ba205ff72ab0eec631559e1","schemaVersion":"ucr.semantic-harvest-receipt/2","semanticObjectHash":"6c9ba1779e67b185368b340a2e00578264209b35cbbcbdccc61bbba9b56868a0","taskId":"multimodel-codex-claude-code-55ce8dd5ed186a51"},"usage":{"cachedInputTokens":87296,"costUsd":null,"inputTokens":100274,"outputTokens":1328,"tokenAccounting":"provider-cli-native","totalTokens":null},"workUsage":{"cachedInputTokens":87296,"costUsd":null,"inputTokens":100274,"outputTokens":1328,"tokenAccounting":"provider-cli-native","totalTokens":null}},"producerClient":"codex","producerFamily":"openai","producerModel":"gpt-5.6-sol","runtime":{"attestationFailed":false,"attested":false,"consumerMcpExposed":false,"correct":true,"costLedger":{"attributionComplete":true,"entries":[{"accountingMethod":"consumer-tools-list:tiktoken","detail":{"exposedTools":0},"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"schema","toolCalls":0},{"accountingMethod":"provider-cli-native","detail":{"additionalModelCalls":0,"semanticDeltaTokens":271},"includedInTotal":true,"inputTokens":100274,"latencyMs":41694.7831,"modelCalls":1,"outputTokens":1328,"phase":"capture","toolCalls":0},{"accountingMethod":"host-preaction-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":135,"modelCalls":0,"outputTokens":0,"phase":"retrieval","toolCalls":1},{"accountingMethod":"trusted-context:utf8-length-div-4","detail":{"alreadyIncludedInConsumerInput":true,"capsuleTokens":59},"includedInTotal":false,"inputTokens":80,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"injection","toolCalls":0},{"accountingMethod":"provider-cli-native-logical-input","detail":null,"includedInTotal":true,"inputTokens":397966,"latencyMs":35043,"modelCalls":1,"outputTokens":1811,"phase":"consumer","toolCalls":0},{"accountingMethod":"independent-grader-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":280.8348999999871,"modelCalls":0,"outputTokens":0,"phase":"validation","toolCalls":2}],"ledgerHash":"266de94c6c997a76d771b5e3681c07a860370b23c8494be9a43a96245fb370fb","missingPhases":[],"runId":"multimodel-codex-claude-code-55ce8dd5ed186a51:runtime","schemaVersion":"ucr.cognitive-cost-ledger/1","totals":{"inputTokens":498240,"latencyMs":77153.61799999999,"modelCalls":2,"outputTokens":3139,"toolCalls":3,"totalTokens":501379}},"delivered":true,"deliveryPhase":"adapter-pre-action","exitCode":0,"firstSuccessorCost":{"additionalModelCalls":0,"captureTokens":271,"hostCaptureLatencyMs":266.78310000000056},"latencyMs":35043,"mistakeExecuted":false,"nativeGuardAuditEvents":0,"nativeGuardEnforced":false,"nativeGuardWired":true,"outcomeGrade":{"canonicalSourceChanged":true,"generatedHash":"45d0fc7d9154fea2cbd56bca7a92a636401fcf462ec1a8314c13856eee5d9459","generatedSynchronized":true,"latencyMs":137.98499999998603,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"5287aef92b0b0f49a0115d383be641640017d7a41325b738559360825cdb856e","verifierExitCode":0,"verifierOutput":"verification passed"},"pipelineUsage":{"inputTokens":498240,"latencyMs":77153.61799999999,"modelCalls":2,"outputTokens":3139,"tokenAccounting":"phase-ledger:provider-native","toolCalls":3,"totalTokens":501379},"preActionReceipt":{"action":"deliver","budget":160,"capsuleIds":["capsule:f14a79c15bc14e9f95ef55b6"],"capsuleTokens":59,"consumerMcpExposed":false,"delivered":true,"deliveryEventId":"019fe9e8-6c05-7af7-a259-6aee64cb22a9","exposedTools":[],"injectionChannel":"claude-host-appended-system-prompt","injectionTokens":80,"latencyMs":135,"objectIds":["failure:ca04c11d7e0fcc72edd18062"],"receiptHash":"eb90aa8ef265c07053e1ee4c859774f1c0e25e2562687ac20c80e0dc6d7c9f18","retrievalAttempted":true,"schemaVersion":"ucr.pre-action-receipt/1","sessionId":"consumer-session","staticSchemaTokens":0,"taskId":"multimodel-codex-claude-code-55ce8dd5ed186a51","trigger":"task"},"preflightLatencyMs":135,"retrievalAttempted":true,"selected":true,"usage":{"cachedInputTokens":385003,"capsuleTokens":59,"costUsd":0.22111689999999995,"inputTokens":397966,"instructionTokens":80,"outputTokens":1811,"staticSchemaTokens":0,"tokenAccounting":"provider-cli-native-logical-input","totalTokens":399777}},"schemaVersion":"ucr.multimodel-attempt/1","study":"cross-client-handoff","taskHash":"83bfcba450dd91331fd4933223a07114e8d04744a7b886ec1f59a0963a3c1bf4","transcriptPublished":false} {"attemptHash":"cd94e1edd4eb7d45c27621d02946270fd9d57a92d47228b6c39410f398218299","canonicalEvents":5,"consumerClient":"claude-code","consumerFamily":"anthropic","consumerModel":"sonnet","control":{"abstained":false,"correct":true,"costLedger":{"attributionComplete":true,"entries":[{"accountingMethod":"consumer-tools-list:tiktoken","detail":{"exposedTools":0},"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"schema","toolCalls":0},{"accountingMethod":"provider-cli-native","detail":{"additionalModelCalls":0,"semanticDeltaTokens":0},"includedInTotal":true,"inputTokens":117227,"latencyMs":47234,"modelCalls":1,"outputTokens":1116,"phase":"capture","toolCalls":0},{"accountingMethod":"host-preaction-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"retrieval","toolCalls":0},{"accountingMethod":"trusted-context:utf8-length-div-4","detail":{"alreadyIncludedInConsumerInput":true,"capsuleTokens":0},"includedInTotal":false,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"injection","toolCalls":0},{"accountingMethod":"provider-cli-native-logical-input","detail":null,"includedInTotal":true,"inputTokens":356243,"latencyMs":25987,"modelCalls":1,"outputTokens":1715,"phase":"consumer","toolCalls":0},{"accountingMethod":"independent-grader-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":282.72200000000885,"modelCalls":0,"outputTokens":0,"phase":"validation","toolCalls":2}],"ledgerHash":"5614e71ed7849476dabff9e6c094352812cfee575d042d501ff9154f9d806b84","missingPhases":[],"runId":"multimodel-codex-claude-code-fbe7042afe1bda85:control","schemaVersion":"ucr.cognitive-cost-ledger/1","totals":{"inputTokens":473470,"latencyMs":73503.72200000001,"modelCalls":2,"outputTokens":2831,"toolCalls":2,"totalTokens":476301}},"exitCode":0,"latencyMs":25987,"mistakeExecuted":false,"outcomeGrade":{"canonicalSourceChanged":true,"generatedHash":"268cd25e33474294d7ee092605412dd0b70ac127cbcb3be99d2d0f9fc8dcd0b6","generatedSynchronized":true,"latencyMs":140.94940000001225,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"ade16e63dac5269eb9edb98eca42051d9bdcb45baf5f7384e9006a730a92edca","verifierExitCode":0,"verifierOutput":"verification passed"},"pipelineUsage":{"inputTokens":473470,"latencyMs":73503.72200000001,"modelCalls":2,"outputTokens":2831,"tokenAccounting":"phase-ledger:provider-native","toolCalls":2,"totalTokens":476301},"predecessorLatencyMs":47234,"predecessorUsage":{"cachedInputTokens":102400,"costUsd":null,"inputTokens":117227,"outputTokens":1116,"tokenAccounting":"provider-cli-native","totalTokens":null},"usage":{"cachedInputTokens":343975,"costUsd":0.20320549999999998,"inputTokens":356243,"outputTokens":1715,"tokenAccounting":"provider-cli-native-logical-input","totalTokens":357958}},"direction":"codex->claude-code","eventStreamHash":"8c982091cae716f208b2cd5367d8bb5e86be2d14dc745e07312e4236ad71b8a4","passed":true,"producer":{"additionalModelCalls":0,"authoredDuringWorkTurn":true,"evidenceAuthenticatedBeforeActivation":true,"exitCode":0,"guardMaterialized":true,"harvestHostLatencyMs":271.3564000000042,"harvestUsage":{"inputTokens":0,"modelCalls":0,"outputTokens":274,"tokenAccounting":"model-output-delta:tiktoken-compatible-estimate","totalTokens":274},"latencyMs":45478,"modelAuthored":true,"predecessorCorrectionVerified":true,"predecessorEvidence":{"attemptedAction":"apply implementation choice EDIT-GENERATED-CLIENT-HOOK","auditHash":"f5dce5e6d60e3c280823d0e601d8764150fb6f793cf30dc644b88d74146f8559","correctedAttemptAccepted":true,"correction":"apply verified implementation choice EDIT-HOOKS-CORE-THEN-SYNC","correctionObserved":true,"expectedOutcome":"task multimodel-codex-claude-code-fbe7042afe1bda85 updates hooks-core, regenerates clients, and does not repeat EDIT-GENERATED-CLIENT-HOOK","finalGrade":{"canonicalSourceChanged":true,"generatedHash":"268cd25e33474294d7ee092605412dd0b70ac127cbcb3be99d2d0f9fc8dcd0b6","generatedSynchronized":true,"latencyMs":140.03579999999783,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"ade16e63dac5269eb9edb98eca42051d9bdcb45baf5f7384e9006a730a92edca","verifierExitCode":0,"verifierOutput":"verification passed"},"observedFailure":"regeneration verifier rejected EDIT-GENERATED-CLIENT-HOOK after overwriting the direct generated-hook edit","rootCause":"client hook copies are generated from hooks-core and direct edits do not survive synchronization","staleAttemptObserved":true,"trigger":"continuing implementation task multimodel-codex-claude-code-fbe7042afe1bda85","verificationEvidence":"external audit observed failed regeneration followed by a passing end-state grade for EDIT-HOOKS-CORE-THEN-SYNC"},"predecessorMistakeObserved":true,"recorded":true,"semanticHarvestReceipt":{"activatedObjectId":"failure:5f81323c28aea4ba7a3e2eca","additionalModelCalls":0,"authoredDuringWorkTurn":true,"deltaTokens":274,"eventIds":["019fe9eb-f8bb-75ce-92e3-a704c0f6ce68","019fe9eb-f8bc-7e39-9e3a-c9b2a0bc7731","019fe9eb-f8bd-71c0-8529-5c19858d5b73","019fe9eb-f8bd-7e4a-9b61-84f4750a2bf0"],"evidenceAuthenticatedBeforeActivation":true,"hostLatencyMs":271.3564000000042,"kind":"failure","modelAuthored":true,"modelOutputHash":"a4c066b3ab2dbef7f4e8dd4a47339748e9f16b2bc1a70d0cd27a598624afeb47","receiptHash":"3f8ce4fde68a7ae303711b97c67572a907ad89292a5909d64c9d3d0d6c4890a6","schemaVersion":"ucr.semantic-harvest-receipt/2","semanticObjectHash":"b2192cebd34d2a418f85313106402208925b29bff95fb082dc72e4d8b396e8d8","taskId":"multimodel-codex-claude-code-fbe7042afe1bda85"},"usage":{"cachedInputTokens":70144,"costUsd":null,"inputTokens":85268,"outputTokens":1313,"tokenAccounting":"provider-cli-native","totalTokens":null},"workUsage":{"cachedInputTokens":70144,"costUsd":null,"inputTokens":85268,"outputTokens":1313,"tokenAccounting":"provider-cli-native","totalTokens":null}},"producerClient":"codex","producerFamily":"openai","producerModel":"gpt-5.6-sol","runtime":{"attestationFailed":false,"attested":false,"consumerMcpExposed":false,"correct":true,"costLedger":{"attributionComplete":true,"entries":[{"accountingMethod":"consumer-tools-list:tiktoken","detail":{"exposedTools":0},"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"schema","toolCalls":0},{"accountingMethod":"provider-cli-native","detail":{"additionalModelCalls":0,"semanticDeltaTokens":274},"includedInTotal":true,"inputTokens":85268,"latencyMs":45749.356400000004,"modelCalls":1,"outputTokens":1313,"phase":"capture","toolCalls":0},{"accountingMethod":"host-preaction-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":134,"modelCalls":0,"outputTokens":0,"phase":"retrieval","toolCalls":1},{"accountingMethod":"trusted-context:utf8-length-div-4","detail":{"alreadyIncludedInConsumerInput":true,"capsuleTokens":64},"includedInTotal":false,"inputTokens":85,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"injection","toolCalls":0},{"accountingMethod":"provider-cli-native-logical-input","detail":null,"includedInTotal":true,"inputTokens":197310,"latencyMs":20672,"modelCalls":1,"outputTokens":1303,"phase":"consumer","toolCalls":0},{"accountingMethod":"independent-grader-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":276.71970000000147,"modelCalls":0,"outputTokens":0,"phase":"validation","toolCalls":2}],"ledgerHash":"ce8a32f53924ba6482c98c584fbf444c5eff675c01db072e38a35c5995b3eb6a","missingPhases":[],"runId":"multimodel-codex-claude-code-fbe7042afe1bda85:runtime","schemaVersion":"ucr.cognitive-cost-ledger/1","totals":{"inputTokens":282578,"latencyMs":66832.0761,"modelCalls":2,"outputTokens":2616,"toolCalls":3,"totalTokens":285194}},"delivered":true,"deliveryPhase":"adapter-pre-action","exitCode":0,"firstSuccessorCost":{"additionalModelCalls":0,"captureTokens":274,"hostCaptureLatencyMs":271.3564000000042},"latencyMs":20672,"mistakeExecuted":false,"nativeGuardAuditEvents":0,"nativeGuardEnforced":false,"nativeGuardWired":true,"outcomeGrade":{"canonicalSourceChanged":true,"generatedHash":"268cd25e33474294d7ee092605412dd0b70ac127cbcb3be99d2d0f9fc8dcd0b6","generatedSynchronized":true,"latencyMs":136.68390000000363,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"ade16e63dac5269eb9edb98eca42051d9bdcb45baf5f7384e9006a730a92edca","verifierExitCode":0,"verifierOutput":"verification passed"},"pipelineUsage":{"inputTokens":282578,"latencyMs":66832.0761,"modelCalls":2,"outputTokens":2616,"tokenAccounting":"phase-ledger:provider-native","toolCalls":3,"totalTokens":285194},"preActionReceipt":{"action":"deliver","budget":160,"capsuleIds":["capsule:f8d1baf4ab92ad0ba79babf9"],"capsuleTokens":64,"consumerMcpExposed":false,"delivered":true,"deliveryEventId":"019fe9eb-f94d-7015-bacf-e76eb98fdcbe","exposedTools":[],"injectionChannel":"claude-host-appended-system-prompt","injectionTokens":85,"latencyMs":134,"objectIds":["failure:5f81323c28aea4ba7a3e2eca"],"receiptHash":"c9f9e7dd00d9657ad57ce1def83c642177f49a6d2ef9ddef9897d69835a350cf","retrievalAttempted":true,"schemaVersion":"ucr.pre-action-receipt/1","sessionId":"consumer-session","staticSchemaTokens":0,"taskId":"multimodel-codex-claude-code-fbe7042afe1bda85","trigger":"task"},"preflightLatencyMs":134,"retrievalAttempted":true,"selected":true,"usage":{"cachedInputTokens":185593,"capsuleTokens":64,"costUsd":0.1462389,"inputTokens":197310,"instructionTokens":85,"outputTokens":1303,"staticSchemaTokens":0,"tokenAccounting":"provider-cli-native-logical-input","totalTokens":198613}},"schemaVersion":"ucr.multimodel-attempt/1","study":"cross-client-handoff","taskHash":"f385bfc14873249bb0d00a1061fabb7a4669bd3e40520041a367490cc813193c","transcriptPublished":false} {"attemptHash":"1d089542bb210987ba2a00c3755ef9d700faa24258ed4e3702a43f8b4caec135","canonicalEvents":5,"consumerClient":"codex","consumerFamily":"openai","consumerModel":"gpt-5.6-sol","control":{"abstained":false,"correct":true,"costLedger":{"attributionComplete":true,"entries":[{"accountingMethod":"consumer-tools-list:tiktoken","detail":{"exposedTools":0},"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"schema","toolCalls":0},{"accountingMethod":"provider-cli-native-logical-input","detail":{"additionalModelCalls":0,"semanticDeltaTokens":0},"includedInTotal":true,"inputTokens":282292,"latencyMs":28919,"modelCalls":1,"outputTokens":2074,"phase":"capture","toolCalls":0},{"accountingMethod":"host-preaction-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"retrieval","toolCalls":0},{"accountingMethod":"trusted-context:utf8-length-div-4","detail":{"alreadyIncludedInConsumerInput":true,"capsuleTokens":0},"includedInTotal":false,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"injection","toolCalls":0},{"accountingMethod":"provider-cli-native","detail":null,"includedInTotal":true,"inputTokens":117151,"latencyMs":85956,"modelCalls":1,"outputTokens":1439,"phase":"consumer","toolCalls":0},{"accountingMethod":"independent-grader-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":283.3888999999799,"modelCalls":0,"outputTokens":0,"phase":"validation","toolCalls":2}],"ledgerHash":"b0baf3965fbadc92e882cff2ba9fa2fec415bec30366e1c3854e185b7b701c16","missingPhases":[],"runId":"multimodel-claude-code-codex-94d5f201e72a01a6:control","schemaVersion":"ucr.cognitive-cost-ledger/1","totals":{"inputTokens":399443,"latencyMs":115158.38889999998,"modelCalls":2,"outputTokens":3513,"toolCalls":2,"totalTokens":402956}},"exitCode":0,"latencyMs":85956,"mistakeExecuted":true,"outcomeGrade":{"canonicalSourceChanged":true,"generatedHash":"ec337d64a59ed22cb1310d6dd1a8a4a0b709ee19cc3dc9d6b04698306d9268d8","generatedSynchronized":true,"latencyMs":138.83479999998235,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"073aa426dffc0c0530caed5338626ce1463d33210f509f2a3128ba2430a8fc82","verifierExitCode":0,"verifierOutput":"verification passed"},"pipelineUsage":{"inputTokens":399443,"latencyMs":115158.38889999998,"modelCalls":2,"outputTokens":3513,"tokenAccounting":"phase-ledger:provider-native","toolCalls":2,"totalTokens":402956},"predecessorLatencyMs":28919,"predecessorUsage":{"cachedInputTokens":268984,"costUsd":0.1924142,"inputTokens":282292,"outputTokens":2074,"tokenAccounting":"provider-cli-native-logical-input","totalTokens":284366},"usage":{"cachedInputTokens":100352,"costUsd":null,"inputTokens":117151,"outputTokens":1439,"tokenAccounting":"provider-cli-native","totalTokens":null}},"direction":"claude-code->codex","eventStreamHash":"e690e86057f1d820430a48f643873232ddbdc892e4549ea65ec9547e61ecef22","passed":true,"producer":{"additionalModelCalls":0,"authoredDuringWorkTurn":true,"evidenceAuthenticatedBeforeActivation":true,"exitCode":0,"guardMaterialized":true,"harvestHostLatencyMs":268.05429999998887,"harvestUsage":{"inputTokens":0,"modelCalls":0,"outputTokens":311,"tokenAccounting":"model-output-delta:tiktoken-compatible-estimate","totalTokens":311},"latencyMs":36627,"modelAuthored":true,"predecessorCorrectionVerified":true,"predecessorEvidence":{"attemptedAction":"apply implementation choice EDIT-GENERATED-CLIENT-HOOK","auditHash":"f5dce5e6d60e3c280823d0e601d8764150fb6f793cf30dc644b88d74146f8559","correctedAttemptAccepted":true,"correction":"apply verified implementation choice EDIT-HOOKS-CORE-THEN-SYNC","correctionObserved":true,"expectedOutcome":"task multimodel-claude-code-codex-94d5f201e72a01a6 updates hooks-core, regenerates clients, and does not repeat EDIT-GENERATED-CLIENT-HOOK","finalGrade":{"canonicalSourceChanged":true,"generatedHash":"ec337d64a59ed22cb1310d6dd1a8a4a0b709ee19cc3dc9d6b04698306d9268d8","generatedSynchronized":true,"latencyMs":143.37679999999818,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"073aa426dffc0c0530caed5338626ce1463d33210f509f2a3128ba2430a8fc82","verifierExitCode":0,"verifierOutput":"verification passed"},"observedFailure":"regeneration verifier rejected EDIT-GENERATED-CLIENT-HOOK after overwriting the direct generated-hook edit","rootCause":"client hook copies are generated from hooks-core and direct edits do not survive synchronization","staleAttemptObserved":true,"trigger":"continuing implementation task multimodel-claude-code-codex-94d5f201e72a01a6","verificationEvidence":"external audit observed failed regeneration followed by a passing end-state grade for EDIT-HOOKS-CORE-THEN-SYNC"},"predecessorMistakeObserved":true,"recorded":true,"semanticHarvestReceipt":{"activatedObjectId":"failure:176f132291364da8b5b87e08","additionalModelCalls":0,"authoredDuringWorkTurn":true,"deltaTokens":311,"eventIds":["019fe9f1-10ac-723b-a071-afeaca9ecd54","019fe9f1-10ae-7c7e-9fbf-a7d0422928c6","019fe9f1-10ae-769c-911c-cb040f9a47d7","019fe9f1-10af-7165-bcff-2da5ef7cbbca"],"evidenceAuthenticatedBeforeActivation":true,"hostLatencyMs":268.05429999998887,"kind":"failure","modelAuthored":true,"modelOutputHash":"f4a54279be90c117df80ffe6ebe2b24067936da63c80d97146762a275bb1efde","receiptHash":"3b1fbe74bb82c6b52b907289b11f70611ef9df8641f7ffddfa90b979caa7d0db","schemaVersion":"ucr.semantic-harvest-receipt/2","semanticObjectHash":"53ba281e86ecaac0fc7f4152966a0699bf61e6d07e89cb2e3688c0ce375443b1","taskId":"multimodel-claude-code-codex-94d5f201e72a01a6"},"usage":{"cachedInputTokens":272521,"costUsd":0.20298130000000003,"inputTokens":286385,"outputTokens":2459,"tokenAccounting":"provider-cli-native-logical-input","totalTokens":288844},"workUsage":{"cachedInputTokens":272521,"costUsd":0.20298130000000003,"inputTokens":286385,"outputTokens":2459,"tokenAccounting":"provider-cli-native-logical-input","totalTokens":288844}},"producerClient":"claude-code","producerFamily":"anthropic","producerModel":"sonnet","runtime":{"attestationFailed":false,"attested":false,"consumerMcpExposed":false,"correct":true,"costLedger":{"attributionComplete":true,"entries":[{"accountingMethod":"consumer-tools-list:tiktoken","detail":{"exposedTools":0},"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"schema","toolCalls":0},{"accountingMethod":"provider-cli-native-logical-input","detail":{"additionalModelCalls":0,"semanticDeltaTokens":311},"includedInTotal":true,"inputTokens":286385,"latencyMs":36895.05429999999,"modelCalls":1,"outputTokens":2459,"phase":"capture","toolCalls":0},{"accountingMethod":"host-preaction-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":136,"modelCalls":0,"outputTokens":0,"phase":"retrieval","toolCalls":1},{"accountingMethod":"trusted-context:utf8-length-div-4","detail":{"alreadyIncludedInConsumerInput":true,"capsuleTokens":91},"includedInTotal":false,"inputTokens":111,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"injection","toolCalls":0},{"accountingMethod":"provider-cli-native","detail":null,"includedInTotal":true,"inputTokens":118891,"latencyMs":49877,"modelCalls":1,"outputTokens":1196,"phase":"consumer","toolCalls":0},{"accountingMethod":"independent-grader-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":280.10680000000866,"modelCalls":0,"outputTokens":0,"phase":"validation","toolCalls":2}],"ledgerHash":"4f8c5debf2cc34df8184146ca40be67d52a0c4b2e70e63857c9a3a8e7f4a0d77","missingPhases":[],"runId":"multimodel-claude-code-codex-94d5f201e72a01a6:runtime","schemaVersion":"ucr.cognitive-cost-ledger/1","totals":{"inputTokens":405276,"latencyMs":87188.1611,"modelCalls":2,"outputTokens":3655,"toolCalls":3,"totalTokens":408931}},"delivered":true,"deliveryPhase":"adapter-pre-action","exitCode":0,"firstSuccessorCost":{"additionalModelCalls":0,"captureTokens":311,"hostCaptureLatencyMs":268.05429999998887},"latencyMs":49877,"mistakeExecuted":false,"nativeGuardAuditEvents":1,"nativeGuardEnforced":true,"nativeGuardWired":true,"outcomeGrade":{"canonicalSourceChanged":true,"generatedHash":"ec337d64a59ed22cb1310d6dd1a8a4a0b709ee19cc3dc9d6b04698306d9268d8","generatedSynchronized":true,"latencyMs":136.73000000001048,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"073aa426dffc0c0530caed5338626ce1463d33210f509f2a3128ba2430a8fc82","verifierExitCode":0,"verifierOutput":"verification passed"},"pipelineUsage":{"inputTokens":405276,"latencyMs":87188.1611,"modelCalls":2,"outputTokens":3655,"tokenAccounting":"phase-ledger:provider-native","toolCalls":3,"totalTokens":408931},"preActionReceipt":{"action":"deliver","budget":160,"capsuleIds":["capsule:e05fd0de789b9b4013e6c3d2"],"capsuleTokens":91,"consumerMcpExposed":false,"delivered":true,"deliveryEventId":"019fe9f1-113e-7b24-9351-811b6b96d16d","exposedTools":[],"injectionChannel":"codex-developer-instructions","injectionTokens":111,"latencyMs":136,"objectIds":["failure:176f132291364da8b5b87e08"],"receiptHash":"2708200aca1dc73d6445531954fe4de1d9faf3494cf206a8a1e5f15f032a5b4e","retrievalAttempted":true,"schemaVersion":"ucr.pre-action-receipt/1","sessionId":"consumer-session","staticSchemaTokens":0,"taskId":"multimodel-claude-code-codex-94d5f201e72a01a6","trigger":"task"},"preflightLatencyMs":136,"retrievalAttempted":true,"selected":true,"usage":{"cachedInputTokens":102400,"capsuleTokens":91,"costUsd":null,"inputTokens":118891,"instructionTokens":111,"outputTokens":1196,"staticSchemaTokens":0,"tokenAccounting":"provider-cli-native","totalTokens":null}},"schemaVersion":"ucr.multimodel-attempt/1","study":"cross-client-handoff","taskHash":"6e749682921b7d93791bdf8ec696789d101378c42a299c93c0bfbd97fe27e2e2","transcriptPublished":false} +{"attemptHash":"6b6a858338484cfacb48c84d61d865183a30ed7405ead87f298a6a3981492aef","canonicalEvents":5,"consumerClient":"claude-code","consumerFamily":"anthropic","consumerModel":"sonnet","control":{"abstained":false,"correct":true,"costLedger":{"attributionComplete":true,"entries":[{"accountingMethod":"consumer-tools-list:tiktoken","detail":{"exposedTools":0},"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"schema","toolCalls":0},{"accountingMethod":"provider-cli-native","detail":{"additionalModelCalls":0,"semanticDeltaTokens":0},"includedInTotal":true,"inputTokens":85261,"latencyMs":40354,"modelCalls":1,"outputTokens":1032,"phase":"capture","toolCalls":0},{"accountingMethod":"host-preaction-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"retrieval","toolCalls":0},{"accountingMethod":"trusted-context:utf8-length-div-4","detail":{"alreadyIncludedInConsumerInput":true,"capsuleTokens":0},"includedInTotal":false,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"injection","toolCalls":0},{"accountingMethod":"provider-cli-native-logical-input","detail":null,"includedInTotal":true,"inputTokens":236744,"latencyMs":26857,"modelCalls":1,"outputTokens":1849,"phase":"consumer","toolCalls":0},{"accountingMethod":"independent-grader-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":305.1478000000061,"modelCalls":0,"outputTokens":0,"phase":"validation","toolCalls":2}],"ledgerHash":"8a9435feb6bf342e75159033c611d77fa6672de6de33cd9a3c29492b5668f4ef","missingPhases":[],"runId":"multimodel-codex-claude-code-7307ea21f2552640:control","schemaVersion":"ucr.cognitive-cost-ledger/1","totals":{"inputTokens":322005,"latencyMs":67516.1478,"modelCalls":2,"outputTokens":2881,"toolCalls":2,"totalTokens":324886}},"exitCode":0,"latencyMs":26857,"mistakeExecuted":false,"outcomeGrade":{"canonicalSourceChanged":true,"generatedHash":"bf047ff6229c44ecc59db82079f7a0bede2323c2668293c9bbcfb5a84ad3ae6e","generatedSynchronized":true,"latencyMs":160.2514000000083,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"bfe31a9f5cb51d6096480a0b507e9ddab94db38ff6f40e06b62f49c32d5a835d","verifierExitCode":0,"verifierOutput":"verification passed"},"pipelineUsage":{"inputTokens":322005,"latencyMs":67516.1478,"modelCalls":2,"outputTokens":2881,"tokenAccounting":"phase-ledger:provider-native","toolCalls":2,"totalTokens":324886},"predecessorLatencyMs":40354,"predecessorUsage":{"cachedInputTokens":77312,"costUsd":null,"inputTokens":85261,"outputTokens":1032,"tokenAccounting":"provider-cli-native","totalTokens":null},"usage":{"cachedInputTokens":224438,"costUsd":0.16960240000000001,"inputTokens":236744,"outputTokens":1849,"tokenAccounting":"provider-cli-native-logical-input","totalTokens":238593}},"direction":"codex->claude-code","eventStreamHash":"3fd304480a3ade88b0bc806231d3115943f63b1ebacfd28ba19bb4e3da048f71","passed":true,"producer":{"additionalModelCalls":0,"authoredDuringWorkTurn":true,"evidenceAuthenticatedBeforeActivation":true,"exitCode":0,"guardMaterialized":true,"harvestHostLatencyMs":283.8990000000049,"harvestUsage":{"inputTokens":0,"modelCalls":0,"outputTokens":267,"tokenAccounting":"model-output-delta:tiktoken-compatible-estimate","totalTokens":267},"latencyMs":45493,"modelAuthored":true,"predecessorCorrectionVerified":true,"predecessorEvidence":{"attemptedAction":"apply implementation choice EDIT-GENERATED-CLIENT-HOOK","auditHash":"f5dce5e6d60e3c280823d0e601d8764150fb6f793cf30dc644b88d74146f8559","correctedAttemptAccepted":true,"correction":"apply verified implementation choice EDIT-HOOKS-CORE-THEN-SYNC","correctionObserved":true,"expectedOutcome":"task multimodel-codex-claude-code-7307ea21f2552640 updates hooks-core, regenerates clients, and does not repeat EDIT-GENERATED-CLIENT-HOOK","finalGrade":{"canonicalSourceChanged":true,"generatedHash":"bf047ff6229c44ecc59db82079f7a0bede2323c2668293c9bbcfb5a84ad3ae6e","generatedSynchronized":true,"latencyMs":155.4212000000116,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"bfe31a9f5cb51d6096480a0b507e9ddab94db38ff6f40e06b62f49c32d5a835d","verifierExitCode":0,"verifierOutput":"verification passed"},"observedFailure":"regeneration verifier rejected EDIT-GENERATED-CLIENT-HOOK after overwriting the direct generated-hook edit","rootCause":"client hook copies are generated from hooks-core and direct edits do not survive synchronization","staleAttemptObserved":true,"trigger":"continuing implementation task multimodel-codex-claude-code-7307ea21f2552640","verificationEvidence":"external audit observed failed regeneration followed by a passing end-state grade for EDIT-HOOKS-CORE-THEN-SYNC"},"predecessorMistakeObserved":true,"recorded":true,"semanticHarvestReceipt":{"activatedObjectId":"failure:9182fa04929ed0b7f818dcc0","additionalModelCalls":0,"authoredDuringWorkTurn":true,"deltaTokens":267,"eventIds":["019febae-76b3-7ab5-997d-b54b535df7aa","019febae-76b4-7ab4-a2b2-7f18eb07981c","019febae-76b5-75fb-8109-8f7ad3ef392b","019febae-76b5-7c1d-bb52-86ee783ad60c"],"evidenceAuthenticatedBeforeActivation":true,"hostLatencyMs":283.8990000000049,"kind":"failure","modelAuthored":true,"modelOutputHash":"ccb83805d470c4653615bbb8c68cebaae8d10db85aca7aec7a4c1ace446ec25b","receiptHash":"9ebf42b6a45b8cec374172411239fecaf5464e5337e843bfd0ae747f2611132a","schemaVersion":"ucr.semantic-harvest-receipt/2","semanticObjectHash":"83b3d7741f9bdf7ac012e4ec3035907e50914d22da67c078528b133b89791600","taskId":"multimodel-codex-claude-code-7307ea21f2552640"},"usage":{"cachedInputTokens":74240,"costUsd":null,"inputTokens":85607,"outputTokens":1289,"tokenAccounting":"provider-cli-native","totalTokens":null},"workUsage":{"cachedInputTokens":74240,"costUsd":null,"inputTokens":85607,"outputTokens":1289,"tokenAccounting":"provider-cli-native","totalTokens":null}},"producerClient":"codex","producerFamily":"openai","producerModel":"gpt-5.6-sol","runtime":{"attestationFailed":false,"attested":false,"consumerMcpExposed":false,"correct":true,"costLedger":{"attributionComplete":true,"entries":[{"accountingMethod":"consumer-tools-list:tiktoken","detail":{"exposedTools":0},"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"schema","toolCalls":0},{"accountingMethod":"provider-cli-native","detail":{"additionalModelCalls":0,"semanticDeltaTokens":267},"includedInTotal":true,"inputTokens":85607,"latencyMs":45776.899000000005,"modelCalls":1,"outputTokens":1289,"phase":"capture","toolCalls":0},{"accountingMethod":"host-preaction-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":145,"modelCalls":0,"outputTokens":0,"phase":"retrieval","toolCalls":1},{"accountingMethod":"trusted-context:utf8-length-div-4","detail":{"alreadyIncludedInConsumerInput":true,"capsuleTokens":57},"includedInTotal":false,"inputTokens":77,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"injection","toolCalls":0},{"accountingMethod":"provider-cli-native-logical-input","detail":null,"includedInTotal":true,"inputTokens":198736,"latencyMs":28617,"modelCalls":1,"outputTokens":1712,"phase":"consumer","toolCalls":0},{"accountingMethod":"independent-grader-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":314.354800000001,"modelCalls":0,"outputTokens":0,"phase":"validation","toolCalls":2}],"ledgerHash":"30ef63671a07386e377f84ee6756867ce551d980192dc7de76cef22588ff966b","missingPhases":[],"runId":"multimodel-codex-claude-code-7307ea21f2552640:runtime","schemaVersion":"ucr.cognitive-cost-ledger/1","totals":{"inputTokens":284343,"latencyMs":74853.2538,"modelCalls":2,"outputTokens":3001,"toolCalls":3,"totalTokens":287344}},"delivered":true,"deliveryPhase":"adapter-pre-action","exitCode":0,"firstSuccessorCost":{"additionalModelCalls":0,"captureTokens":267,"hostCaptureLatencyMs":283.8990000000049},"latencyMs":28617,"mistakeExecuted":false,"nativeGuardAuditEvents":0,"nativeGuardEnforced":false,"nativeGuardWired":true,"outcomeGrade":{"canonicalSourceChanged":true,"generatedHash":"bf047ff6229c44ecc59db82079f7a0bede2323c2668293c9bbcfb5a84ad3ae6e","generatedSynchronized":true,"latencyMs":158.93359999998938,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"bfe31a9f5cb51d6096480a0b507e9ddab94db38ff6f40e06b62f49c32d5a835d","verifierExitCode":0,"verifierOutput":"verification passed"},"pipelineUsage":{"inputTokens":284343,"latencyMs":74853.2538,"modelCalls":2,"outputTokens":3001,"tokenAccounting":"phase-ledger:provider-native","toolCalls":3,"totalTokens":287344},"preActionReceipt":{"action":"deliver","budget":160,"capsuleIds":["capsule:ba496649483b6b8b6290e61b"],"capsuleTokens":57,"consumerMcpExposed":false,"delivered":true,"deliveryEventId":"019febae-774f-794e-99a3-781d17e90a05","exposedTools":[],"injectionChannel":"claude-host-appended-system-prompt","injectionTokens":77,"latencyMs":145,"objectIds":["failure:9182fa04929ed0b7f818dcc0"],"receiptHash":"c2aa57fdafc2bbaa62717924465589d58823830aa0e3d5a7a365fb0b1ea5bcd8","retrievalAttempted":true,"schemaVersion":"ucr.pre-action-receipt/1","sessionId":"consumer-session","staticSchemaTokens":0,"taskId":"multimodel-codex-claude-code-7307ea21f2552640","trigger":"task"},"preflightLatencyMs":145,"retrievalAttempted":true,"selected":true,"usage":{"cachedInputTokens":186401,"capsuleTokens":57,"costUsd":0.15630629999999998,"inputTokens":198736,"instructionTokens":77,"outputTokens":1712,"staticSchemaTokens":0,"tokenAccounting":"provider-cli-native-logical-input","totalTokens":200448}},"schemaVersion":"ucr.multimodel-attempt/1","study":"cross-client-handoff","taskHash":"b6b79053b282a7ea13364797f5c3c098f9c94e1eec8f6ba2847ef3002e3845cb","transcriptPublished":false} +{"attemptHash":"ec1d517d2f8afad657c0419a0e5076927d3956122496b2b83c9b5ab774ced305","canonicalEvents":5,"consumerClient":"codex","consumerFamily":"openai","consumerModel":"gpt-5.6-sol","control":{"abstained":false,"correct":true,"costLedger":{"attributionComplete":true,"entries":[{"accountingMethod":"consumer-tools-list:tiktoken","detail":{"exposedTools":0},"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"schema","toolCalls":0},{"accountingMethod":"provider-cli-native-logical-input","detail":{"additionalModelCalls":0,"semanticDeltaTokens":0},"includedInTotal":true,"inputTokens":286305,"latencyMs":37708,"modelCalls":1,"outputTokens":2552,"phase":"capture","toolCalls":0},{"accountingMethod":"host-preaction-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"retrieval","toolCalls":0},{"accountingMethod":"trusted-context:utf8-length-div-4","detail":{"alreadyIncludedInConsumerInput":true,"capsuleTokens":0},"includedInTotal":false,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"injection","toolCalls":0},{"accountingMethod":"provider-cli-native","detail":null,"includedInTotal":true,"inputTokens":87615,"latencyMs":40346,"modelCalls":1,"outputTokens":885,"phase":"consumer","toolCalls":0},{"accountingMethod":"independent-grader-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":719.0749000000142,"modelCalls":0,"outputTokens":0,"phase":"validation","toolCalls":2}],"ledgerHash":"7a9a45795117eca7085bc0c46b28c3321c9ed0d817befa7eea04a54d7cfeb6fd","missingPhases":[],"runId":"multimodel-claude-code-codex-b25920ecf84c60f3:control","schemaVersion":"ucr.cognitive-cost-ledger/1","totals":{"inputTokens":373920,"latencyMs":78773.0749,"modelCalls":2,"outputTokens":3437,"toolCalls":2,"totalTokens":377357}},"exitCode":0,"latencyMs":40346,"mistakeExecuted":false,"outcomeGrade":{"canonicalSourceChanged":true,"generatedHash":"d7e2a71156c9a9593402d8b4138dedd25f49e19f2da5388f58c9d8dec9ef9fa4","generatedSynchronized":true,"latencyMs":141.38330000001588,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"17f411dba7fbceba8f0083fd231b07e49e871688a4e80e2f6a3391c61bedf1b5","verifierExitCode":0,"verifierOutput":"verification passed"},"pipelineUsage":{"inputTokens":373920,"latencyMs":78773.0749,"modelCalls":2,"outputTokens":3437,"tokenAccounting":"phase-ledger:provider-native","toolCalls":2,"totalTokens":377357},"predecessorLatencyMs":37708,"predecessorUsage":{"cachedInputTokens":272516,"costUsd":0.20392580000000002,"inputTokens":286305,"outputTokens":2552,"tokenAccounting":"provider-cli-native-logical-input","totalTokens":288857},"usage":{"cachedInputTokens":70144,"costUsd":null,"inputTokens":87615,"outputTokens":885,"tokenAccounting":"provider-cli-native","totalTokens":null}},"direction":"claude-code->codex","eventStreamHash":"8fe978479f50d9c2cfd31215f812250d6a681dd7c24957ef82108894fc94d2c1","passed":true,"producer":{"additionalModelCalls":0,"authoredDuringWorkTurn":true,"evidenceAuthenticatedBeforeActivation":true,"exitCode":0,"guardMaterialized":true,"harvestHostLatencyMs":302.5072999999975,"harvestUsage":{"inputTokens":0,"modelCalls":0,"outputTokens":322,"tokenAccounting":"model-output-delta:tiktoken-compatible-estimate","totalTokens":322},"latencyMs":36061,"modelAuthored":true,"predecessorCorrectionVerified":true,"predecessorEvidence":{"attemptedAction":"apply implementation choice EDIT-GENERATED-CLIENT-HOOK","auditHash":"f5dce5e6d60e3c280823d0e601d8764150fb6f793cf30dc644b88d74146f8559","correctedAttemptAccepted":true,"correction":"apply verified implementation choice EDIT-HOOKS-CORE-THEN-SYNC","correctionObserved":true,"expectedOutcome":"task multimodel-claude-code-codex-b25920ecf84c60f3 updates hooks-core, regenerates clients, and does not repeat EDIT-GENERATED-CLIENT-HOOK","finalGrade":{"canonicalSourceChanged":true,"generatedHash":"d7e2a71156c9a9593402d8b4138dedd25f49e19f2da5388f58c9d8dec9ef9fa4","generatedSynchronized":true,"latencyMs":144.05900000000838,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"17f411dba7fbceba8f0083fd231b07e49e871688a4e80e2f6a3391c61bedf1b5","verifierExitCode":0,"verifierOutput":"verification passed"},"observedFailure":"regeneration verifier rejected EDIT-GENERATED-CLIENT-HOOK after overwriting the direct generated-hook edit","rootCause":"client hook copies are generated from hooks-core and direct edits do not survive synchronization","staleAttemptObserved":true,"trigger":"continuing implementation task multimodel-claude-code-codex-b25920ecf84c60f3","verificationEvidence":"external audit observed failed regeneration followed by a passing end-state grade for EDIT-HOOKS-CORE-THEN-SYNC"},"predecessorMistakeObserved":true,"recorded":true,"semanticHarvestReceipt":{"activatedObjectId":"failure:48a0ab17763ecd1e3d09fbd6","additionalModelCalls":0,"authoredDuringWorkTurn":true,"deltaTokens":322,"eventIds":["019febb3-19af-7591-b0a5-fa36870abe8c","019febb3-19b1-7a52-95df-b4ee618261f7","019febb3-19b2-72c8-b8ba-480e2cbd9340","019febb3-19b2-7d36-bac8-7007a6482160"],"evidenceAuthenticatedBeforeActivation":true,"hostLatencyMs":302.5072999999975,"kind":"failure","modelAuthored":true,"modelOutputHash":"3007971686478acafce9b336de3e1882a949f31c0f7f7b48bb409e18b109d9fd","receiptHash":"7f2feee12030cf9e2f6715cd3e70e40e4673cbeb700c214e8fb7fd9bfb4a9e43","schemaVersion":"ucr.semantic-harvest-receipt/2","semanticObjectHash":"de6e11122523cb723d9e2b6cc942dc5b7c806f91b38164d26d96be988875d68f","taskId":"multimodel-claude-code-codex-b25920ecf84c60f3"},"usage":{"cachedInputTokens":270537,"costUsd":0.19137909999999997,"inputTokens":283538,"outputTokens":2070,"tokenAccounting":"provider-cli-native-logical-input","totalTokens":285608},"workUsage":{"cachedInputTokens":270537,"costUsd":0.19137909999999997,"inputTokens":283538,"outputTokens":2070,"tokenAccounting":"provider-cli-native-logical-input","totalTokens":285608}},"producerClient":"claude-code","producerFamily":"anthropic","producerModel":"sonnet","runtime":{"attestationFailed":false,"attested":false,"consumerMcpExposed":false,"correct":true,"costLedger":{"attributionComplete":true,"entries":[{"accountingMethod":"consumer-tools-list:tiktoken","detail":{"exposedTools":0},"includedInTotal":true,"inputTokens":0,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"schema","toolCalls":0},{"accountingMethod":"provider-cli-native-logical-input","detail":{"additionalModelCalls":0,"semanticDeltaTokens":322},"includedInTotal":true,"inputTokens":283538,"latencyMs":36363.5073,"modelCalls":1,"outputTokens":2070,"phase":"capture","toolCalls":0},{"accountingMethod":"host-preaction-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":163,"modelCalls":0,"outputTokens":0,"phase":"retrieval","toolCalls":1},{"accountingMethod":"trusted-context:utf8-length-div-4","detail":{"alreadyIncludedInConsumerInput":true,"capsuleTokens":98},"includedInTotal":false,"inputTokens":119,"latencyMs":0,"modelCalls":0,"outputTokens":0,"phase":"injection","toolCalls":0},{"accountingMethod":"provider-cli-native","detail":null,"includedInTotal":true,"inputTokens":88367,"latencyMs":41548,"modelCalls":1,"outputTokens":1191,"phase":"consumer","toolCalls":0},{"accountingMethod":"independent-grader-timer","detail":null,"includedInTotal":true,"inputTokens":0,"latencyMs":282.2514000000083,"modelCalls":0,"outputTokens":0,"phase":"validation","toolCalls":2}],"ledgerHash":"412b8024caf92f12433095d4875b78f4c0b5378cace1ac8b96b886cb14e151c4","missingPhases":[],"runId":"multimodel-claude-code-codex-b25920ecf84c60f3:runtime","schemaVersion":"ucr.cognitive-cost-ledger/1","totals":{"inputTokens":371905,"latencyMs":78356.7587,"modelCalls":2,"outputTokens":3261,"toolCalls":3,"totalTokens":375166}},"delivered":true,"deliveryPhase":"adapter-pre-action","exitCode":0,"firstSuccessorCost":{"additionalModelCalls":0,"captureTokens":322,"hostCaptureLatencyMs":302.5072999999975},"latencyMs":41548,"mistakeExecuted":false,"nativeGuardAuditEvents":0,"nativeGuardEnforced":false,"nativeGuardWired":true,"outcomeGrade":{"canonicalSourceChanged":true,"generatedHash":"d7e2a71156c9a9593402d8b4138dedd25f49e19f2da5388f58c9d8dec9ef9fa4","generatedSynchronized":true,"latencyMs":138.1923999999999,"passed":true,"protectedFilesUnchanged":true,"sourceHash":"17f411dba7fbceba8f0083fd231b07e49e871688a4e80e2f6a3391c61bedf1b5","verifierExitCode":0,"verifierOutput":"verification passed"},"pipelineUsage":{"inputTokens":371905,"latencyMs":78356.7587,"modelCalls":2,"outputTokens":3261,"tokenAccounting":"phase-ledger:provider-native","toolCalls":3,"totalTokens":375166},"preActionReceipt":{"action":"deliver","budget":160,"capsuleIds":["capsule:c7f9e4017778ccbd3aa49915"],"capsuleTokens":98,"consumerMcpExposed":false,"delivered":true,"deliveryEventId":"019febb3-1a5b-7eb4-a134-0eca6cd147db","exposedTools":[],"injectionChannel":"codex-developer-instructions","injectionTokens":119,"latencyMs":163,"objectIds":["failure:48a0ab17763ecd1e3d09fbd6"],"receiptHash":"125fbdfb9bda2de29a698828edbb1dbfff61df8a825342a3f19127b7be107225","retrievalAttempted":true,"schemaVersion":"ucr.pre-action-receipt/1","sessionId":"consumer-session","staticSchemaTokens":0,"taskId":"multimodel-claude-code-codex-b25920ecf84c60f3","trigger":"task"},"preflightLatencyMs":163,"retrievalAttempted":true,"selected":true,"usage":{"cachedInputTokens":74240,"capsuleTokens":98,"costUsd":null,"inputTokens":88367,"instructionTokens":119,"outputTokens":1191,"staticSchemaTokens":0,"tokenAccounting":"provider-cli-native","totalTokens":null}},"schemaVersion":"ucr.multimodel-attempt/1","study":"cross-client-handoff","taskHash":"883cbbf83125f4efe08b01ea058c0f38fe0c1ef6f858d019fac14ef4f593c9a0","transcriptPublished":false} diff --git a/evals/ucr/results/stateful-matched-claude-to-codex-v1.json b/evals/ucr/results/stateful-matched-claude-to-codex-v1.json new file mode 100644 index 000000000..83cebf13c --- /dev/null +++ b/evals/ucr/results/stateful-matched-claude-to-codex-v1.json @@ -0,0 +1,893 @@ +{ + "schemaVersion": "ucr.live-multimodel-handoff/1", + "evidenceClass": "live-multimodel-executable-smoke-not-powered-effectiveness", + "executedAt": "2026-08-10T12:44:14.819Z", + "sourceTreeHash": "96fcf325cd1945645579e83c48f0164c569017fdf952546ebe174f225c24df6d", + "requestedDirections": [ + "claude-code->codex" + ], + "directionsPassed": 1, + "directionResults": [ + { + "study": "cross-client-handoff", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "sonnet", + "consumerModel": "gpt-5.6-sol", + "taskHash": "883cbbf83125f4efe08b01ea058c0f38fe0c1ef6f858d019fac14ef4f593c9a0", + "control": { + "exitCode": 0, + "correct": true, + "abstained": false, + "mistakeExecuted": false, + "outcomeGrade": { + "passed": true, + "protectedFilesUnchanged": true, + "canonicalSourceChanged": true, + "generatedSynchronized": true, + "verifierExitCode": 0, + "verifierOutput": "verification passed", + "sourceHash": "17f411dba7fbceba8f0083fd231b07e49e871688a4e80e2f6a3391c61bedf1b5", + "generatedHash": "d7e2a71156c9a9593402d8b4138dedd25f49e19f2da5388f58c9d8dec9ef9fa4", + "latencyMs": 141.38330000001588 + }, + "usage": { + "inputTokens": 87615, + "cachedInputTokens": 70144, + "outputTokens": 885, + "totalTokens": null, + "costUsd": null, + "tokenAccounting": "provider-cli-native" + }, + "latencyMs": 40346, + "predecessorUsage": { + "inputTokens": 286305, + "cachedInputTokens": 272516, + "outputTokens": 2552, + "totalTokens": 288857, + "costUsd": 0.20392580000000002, + "tokenAccounting": "provider-cli-native-logical-input" + }, + "predecessorLatencyMs": 37708, + "costLedger": { + "schemaVersion": "ucr.cognitive-cost-ledger/1", + "runId": "multimodel-claude-code-codex-b25920ecf84c60f3:control", + "entries": [ + { + "phase": "schema", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "consumer-tools-list:tiktoken", + "includedInTotal": true, + "detail": { + "exposedTools": 0 + } + }, + { + "phase": "capture", + "inputTokens": 286305, + "outputTokens": 2552, + "latencyMs": 37708, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native-logical-input", + "includedInTotal": true, + "detail": { + "semanticDeltaTokens": 0, + "additionalModelCalls": 0 + } + }, + { + "phase": "retrieval", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "host-preaction-timer", + "includedInTotal": true, + "detail": null + }, + { + "phase": "injection", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "trusted-context:utf8-length-div-4", + "includedInTotal": false, + "detail": { + "alreadyIncludedInConsumerInput": true, + "capsuleTokens": 0 + } + }, + { + "phase": "consumer", + "inputTokens": 87615, + "outputTokens": 885, + "latencyMs": 40346, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native", + "includedInTotal": true, + "detail": null + }, + { + "phase": "validation", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 719.0749000000142, + "toolCalls": 2, + "modelCalls": 0, + "accountingMethod": "independent-grader-timer", + "includedInTotal": true, + "detail": null + } + ], + "totals": { + "inputTokens": 373920, + "outputTokens": 3437, + "totalTokens": 377357, + "latencyMs": 78773.0749, + "toolCalls": 2, + "modelCalls": 2 + }, + "missingPhases": [], + "attributionComplete": true, + "ledgerHash": "7a9a45795117eca7085bc0c46b28c3321c9ed0d817befa7eea04a54d7cfeb6fd" + }, + "pipelineUsage": { + "inputTokens": 373920, + "outputTokens": 3437, + "totalTokens": 377357, + "latencyMs": 78773.0749, + "toolCalls": 2, + "modelCalls": 2, + "tokenAccounting": "phase-ledger:provider-native" + } + }, + "producer": { + "exitCode": 0, + "recorded": true, + "modelAuthored": true, + "authoredDuringWorkTurn": true, + "evidenceAuthenticatedBeforeActivation": true, + "additionalModelCalls": 0, + "guardMaterialized": true, + "predecessorMistakeObserved": true, + "predecessorCorrectionVerified": true, + "predecessorEvidence": { + "trigger": "continuing implementation task multimodel-claude-code-codex-b25920ecf84c60f3", + "attemptedAction": "apply implementation choice EDIT-GENERATED-CLIENT-HOOK", + "observedFailure": "regeneration verifier rejected EDIT-GENERATED-CLIENT-HOOK after overwriting the direct generated-hook edit", + "rootCause": "client hook copies are generated from hooks-core and direct edits do not survive synchronization", + "correction": "apply verified implementation choice EDIT-HOOKS-CORE-THEN-SYNC", + "verificationEvidence": "external audit observed failed regeneration followed by a passing end-state grade for EDIT-HOOKS-CORE-THEN-SYNC", + "expectedOutcome": "task multimodel-claude-code-codex-b25920ecf84c60f3 updates hooks-core, regenerates clients, and does not repeat EDIT-GENERATED-CLIENT-HOOK", + "staleAttemptObserved": true, + "correctionObserved": true, + "correctedAttemptAccepted": true, + "auditHash": "f5dce5e6d60e3c280823d0e601d8764150fb6f793cf30dc644b88d74146f8559", + "finalGrade": { + "passed": true, + "protectedFilesUnchanged": true, + "canonicalSourceChanged": true, + "generatedSynchronized": true, + "verifierExitCode": 0, + "verifierOutput": "verification passed", + "sourceHash": "17f411dba7fbceba8f0083fd231b07e49e871688a4e80e2f6a3391c61bedf1b5", + "generatedHash": "d7e2a71156c9a9593402d8b4138dedd25f49e19f2da5388f58c9d8dec9ef9fa4", + "latencyMs": 144.05900000000838 + } + }, + "semanticHarvestReceipt": { + "schemaVersion": "ucr.semantic-harvest-receipt/2", + "modelAuthored": true, + "authoredDuringWorkTurn": true, + "evidenceAuthenticatedBeforeActivation": true, + "additionalModelCalls": 0, + "kind": "failure", + "taskId": "multimodel-claude-code-codex-b25920ecf84c60f3", + "deltaTokens": 322, + "hostLatencyMs": 302.5072999999975, + "modelOutputHash": "3007971686478acafce9b336de3e1882a949f31c0f7f7b48bb409e18b109d9fd", + "semanticObjectHash": "de6e11122523cb723d9e2b6cc942dc5b7c806f91b38164d26d96be988875d68f", + "activatedObjectId": "failure:48a0ab17763ecd1e3d09fbd6", + "eventIds": [ + "019febb3-19af-7591-b0a5-fa36870abe8c", + "019febb3-19b1-7a52-95df-b4ee618261f7", + "019febb3-19b2-72c8-b8ba-480e2cbd9340", + "019febb3-19b2-7d36-bac8-7007a6482160" + ], + "receiptHash": "7f2feee12030cf9e2f6715cd3e70e40e4673cbeb700c214e8fb7fd9bfb4a9e43" + }, + "workUsage": { + "inputTokens": 283538, + "cachedInputTokens": 270537, + "outputTokens": 2070, + "totalTokens": 285608, + "costUsd": 0.19137909999999997, + "tokenAccounting": "provider-cli-native-logical-input" + }, + "harvestUsage": { + "inputTokens": 0, + "outputTokens": 322, + "totalTokens": 322, + "modelCalls": 0, + "tokenAccounting": "model-output-delta:tiktoken-compatible-estimate" + }, + "usage": { + "inputTokens": 283538, + "cachedInputTokens": 270537, + "outputTokens": 2070, + "totalTokens": 285608, + "costUsd": 0.19137909999999997, + "tokenAccounting": "provider-cli-native-logical-input" + }, + "latencyMs": 36061, + "harvestHostLatencyMs": 302.5072999999975 + }, + "runtime": { + "exitCode": 0, + "correct": true, + "delivered": true, + "selected": true, + "retrievalAttempted": true, + "attested": false, + "attestationFailed": false, + "deliveryPhase": "adapter-pre-action", + "nativeGuardWired": true, + "nativeGuardEnforced": false, + "nativeGuardAuditEvents": 0, + "mistakeExecuted": false, + "outcomeGrade": { + "passed": true, + "protectedFilesUnchanged": true, + "canonicalSourceChanged": true, + "generatedSynchronized": true, + "verifierExitCode": 0, + "verifierOutput": "verification passed", + "sourceHash": "17f411dba7fbceba8f0083fd231b07e49e871688a4e80e2f6a3391c61bedf1b5", + "generatedHash": "d7e2a71156c9a9593402d8b4138dedd25f49e19f2da5388f58c9d8dec9ef9fa4", + "latencyMs": 138.1923999999999 + }, + "usage": { + "inputTokens": 88367, + "cachedInputTokens": 74240, + "outputTokens": 1191, + "totalTokens": null, + "costUsd": null, + "tokenAccounting": "provider-cli-native", + "staticSchemaTokens": 0, + "capsuleTokens": 98, + "instructionTokens": 119 + }, + "latencyMs": 41548, + "preflightLatencyMs": 163, + "firstSuccessorCost": { + "captureTokens": 322, + "additionalModelCalls": 0, + "hostCaptureLatencyMs": 302.5072999999975 + }, + "costLedger": { + "schemaVersion": "ucr.cognitive-cost-ledger/1", + "runId": "multimodel-claude-code-codex-b25920ecf84c60f3:runtime", + "entries": [ + { + "phase": "schema", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "consumer-tools-list:tiktoken", + "includedInTotal": true, + "detail": { + "exposedTools": 0 + } + }, + { + "phase": "capture", + "inputTokens": 283538, + "outputTokens": 2070, + "latencyMs": 36363.5073, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native-logical-input", + "includedInTotal": true, + "detail": { + "semanticDeltaTokens": 322, + "additionalModelCalls": 0 + } + }, + { + "phase": "retrieval", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 163, + "toolCalls": 1, + "modelCalls": 0, + "accountingMethod": "host-preaction-timer", + "includedInTotal": true, + "detail": null + }, + { + "phase": "injection", + "inputTokens": 119, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "trusted-context:utf8-length-div-4", + "includedInTotal": false, + "detail": { + "alreadyIncludedInConsumerInput": true, + "capsuleTokens": 98 + } + }, + { + "phase": "consumer", + "inputTokens": 88367, + "outputTokens": 1191, + "latencyMs": 41548, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native", + "includedInTotal": true, + "detail": null + }, + { + "phase": "validation", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 282.2514000000083, + "toolCalls": 2, + "modelCalls": 0, + "accountingMethod": "independent-grader-timer", + "includedInTotal": true, + "detail": null + } + ], + "totals": { + "inputTokens": 371905, + "outputTokens": 3261, + "totalTokens": 375166, + "latencyMs": 78356.7587, + "toolCalls": 3, + "modelCalls": 2 + }, + "missingPhases": [], + "attributionComplete": true, + "ledgerHash": "412b8024caf92f12433095d4875b78f4c0b5378cace1ac8b96b886cb14e151c4" + }, + "pipelineUsage": { + "inputTokens": 371905, + "outputTokens": 3261, + "totalTokens": 375166, + "latencyMs": 78356.7587, + "toolCalls": 3, + "modelCalls": 2, + "tokenAccounting": "phase-ledger:provider-native" + }, + "consumerMcpExposed": false, + "preActionReceipt": { + "schemaVersion": "ucr.pre-action-receipt/1", + "retrievalAttempted": true, + "action": "deliver", + "delivered": true, + "taskId": "multimodel-claude-code-codex-b25920ecf84c60f3", + "sessionId": "consumer-session", + "trigger": "task", + "budget": 160, + "capsuleIds": [ + "capsule:c7f9e4017778ccbd3aa49915" + ], + "objectIds": [ + "failure:48a0ab17763ecd1e3d09fbd6" + ], + "capsuleTokens": 98, + "injectionTokens": 119, + "staticSchemaTokens": 0, + "consumerMcpExposed": false, + "exposedTools": [], + "injectionChannel": "codex-developer-instructions", + "deliveryEventId": "019febb3-1a5b-7eb4-a134-0eca6cd147db", + "latencyMs": 163, + "receiptHash": "125fbdfb9bda2de29a698828edbb1dbfff61df8a825342a3f19127b7be107225" + } + }, + "eventStreamHash": "8fe978479f50d9c2cfd31215f812250d6a681dd7c24957ef82108894fc94d2c1", + "canonicalEvents": 5, + "passed": true, + "transcriptPublished": false + } + ], + "clients": { + "codex": { + "family": "openai", + "model": "gpt-5.6-sol", + "version": "codex-cli 0.147.0" + }, + "claude-code": { + "family": "anthropic", + "model": "sonnet", + "version": "2.1.225 (Claude Code)" + }, + "copilot": { + "family": "anthropic", + "model": "claude-sonnet-4.5", + "version": "GitHub Copilot CLI 0.0.396.\nRun 'copilot update' to check for updates." + } + }, + "modelFamilies": 2, + "attestationSurface": { + "profile": "attestation", + "toolNames": [ + "context_receipt_verify" + ], + "schemaTokens": 71, + "tokenAccounting": "tiktoken:cl100k_base" + }, + "controlsAbstained": 0, + "repeatedVerifiedFailures": 0, + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "live-multimodel-70ddcc01aef33de0", + "evidenceClass": "executable-smoke", + "benchmarkHash": "4b2148b8a73d20f9d94a3008e1756f9fb38144818bb527b752ec5ff6ffa06f0e", + "sourceTreeHash": "96fcf325cd1945645579e83c48f0164c569017fdf952546ebe174f225c24df6d", + "runner": { + "name": "live-multimodel-handoff", + "versions": { + "codex": "codex-cli 0.147.0", + "claude-code": "2.1.225 (Claude Code)", + "copilot": "GitHub Copilot CLI 0.0.396.\nRun 'copilot update' to check for updates." + } + }, + "startedAt": "2026-08-10T12:44:14.814Z", + "runHash": "4a5e0f813f2bb5a3237fafadfde21c949daf8b3fb76de2741de694b93751f7b2" + }, + "rows": [ + { + "study": "cross-client-handoff", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "sonnet", + "consumerModel": "gpt-5.6-sol", + "taskHash": "883cbbf83125f4efe08b01ea058c0f38fe0c1ef6f858d019fac14ef4f593c9a0", + "control": { + "exitCode": 0, + "correct": true, + "abstained": false, + "mistakeExecuted": false, + "outcomeGrade": { + "passed": true, + "protectedFilesUnchanged": true, + "canonicalSourceChanged": true, + "generatedSynchronized": true, + "verifierExitCode": 0, + "verifierOutput": "verification passed", + "sourceHash": "17f411dba7fbceba8f0083fd231b07e49e871688a4e80e2f6a3391c61bedf1b5", + "generatedHash": "d7e2a71156c9a9593402d8b4138dedd25f49e19f2da5388f58c9d8dec9ef9fa4", + "latencyMs": 141.38330000001588 + }, + "usage": { + "inputTokens": 87615, + "cachedInputTokens": 70144, + "outputTokens": 885, + "totalTokens": null, + "costUsd": null, + "tokenAccounting": "provider-cli-native" + }, + "latencyMs": 40346, + "predecessorUsage": { + "inputTokens": 286305, + "cachedInputTokens": 272516, + "outputTokens": 2552, + "totalTokens": 288857, + "costUsd": 0.20392580000000002, + "tokenAccounting": "provider-cli-native-logical-input" + }, + "predecessorLatencyMs": 37708, + "costLedger": { + "schemaVersion": "ucr.cognitive-cost-ledger/1", + "runId": "multimodel-claude-code-codex-b25920ecf84c60f3:control", + "entries": [ + { + "phase": "schema", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "consumer-tools-list:tiktoken", + "includedInTotal": true, + "detail": { + "exposedTools": 0 + } + }, + { + "phase": "capture", + "inputTokens": 286305, + "outputTokens": 2552, + "latencyMs": 37708, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native-logical-input", + "includedInTotal": true, + "detail": { + "semanticDeltaTokens": 0, + "additionalModelCalls": 0 + } + }, + { + "phase": "retrieval", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "host-preaction-timer", + "includedInTotal": true, + "detail": null + }, + { + "phase": "injection", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "trusted-context:utf8-length-div-4", + "includedInTotal": false, + "detail": { + "alreadyIncludedInConsumerInput": true, + "capsuleTokens": 0 + } + }, + { + "phase": "consumer", + "inputTokens": 87615, + "outputTokens": 885, + "latencyMs": 40346, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native", + "includedInTotal": true, + "detail": null + }, + { + "phase": "validation", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 719.0749000000142, + "toolCalls": 2, + "modelCalls": 0, + "accountingMethod": "independent-grader-timer", + "includedInTotal": true, + "detail": null + } + ], + "totals": { + "inputTokens": 373920, + "outputTokens": 3437, + "totalTokens": 377357, + "latencyMs": 78773.0749, + "toolCalls": 2, + "modelCalls": 2 + }, + "missingPhases": [], + "attributionComplete": true, + "ledgerHash": "7a9a45795117eca7085bc0c46b28c3321c9ed0d817befa7eea04a54d7cfeb6fd" + }, + "pipelineUsage": { + "inputTokens": 373920, + "outputTokens": 3437, + "totalTokens": 377357, + "latencyMs": 78773.0749, + "toolCalls": 2, + "modelCalls": 2, + "tokenAccounting": "phase-ledger:provider-native" + } + }, + "producer": { + "exitCode": 0, + "recorded": true, + "modelAuthored": true, + "authoredDuringWorkTurn": true, + "evidenceAuthenticatedBeforeActivation": true, + "additionalModelCalls": 0, + "guardMaterialized": true, + "predecessorMistakeObserved": true, + "predecessorCorrectionVerified": true, + "predecessorEvidence": { + "trigger": "continuing implementation task multimodel-claude-code-codex-b25920ecf84c60f3", + "attemptedAction": "apply implementation choice EDIT-GENERATED-CLIENT-HOOK", + "observedFailure": "regeneration verifier rejected EDIT-GENERATED-CLIENT-HOOK after overwriting the direct generated-hook edit", + "rootCause": "client hook copies are generated from hooks-core and direct edits do not survive synchronization", + "correction": "apply verified implementation choice EDIT-HOOKS-CORE-THEN-SYNC", + "verificationEvidence": "external audit observed failed regeneration followed by a passing end-state grade for EDIT-HOOKS-CORE-THEN-SYNC", + "expectedOutcome": "task multimodel-claude-code-codex-b25920ecf84c60f3 updates hooks-core, regenerates clients, and does not repeat EDIT-GENERATED-CLIENT-HOOK", + "staleAttemptObserved": true, + "correctionObserved": true, + "correctedAttemptAccepted": true, + "auditHash": "f5dce5e6d60e3c280823d0e601d8764150fb6f793cf30dc644b88d74146f8559", + "finalGrade": { + "passed": true, + "protectedFilesUnchanged": true, + "canonicalSourceChanged": true, + "generatedSynchronized": true, + "verifierExitCode": 0, + "verifierOutput": "verification passed", + "sourceHash": "17f411dba7fbceba8f0083fd231b07e49e871688a4e80e2f6a3391c61bedf1b5", + "generatedHash": "d7e2a71156c9a9593402d8b4138dedd25f49e19f2da5388f58c9d8dec9ef9fa4", + "latencyMs": 144.05900000000838 + } + }, + "semanticHarvestReceipt": { + "schemaVersion": "ucr.semantic-harvest-receipt/2", + "modelAuthored": true, + "authoredDuringWorkTurn": true, + "evidenceAuthenticatedBeforeActivation": true, + "additionalModelCalls": 0, + "kind": "failure", + "taskId": "multimodel-claude-code-codex-b25920ecf84c60f3", + "deltaTokens": 322, + "hostLatencyMs": 302.5072999999975, + "modelOutputHash": "3007971686478acafce9b336de3e1882a949f31c0f7f7b48bb409e18b109d9fd", + "semanticObjectHash": "de6e11122523cb723d9e2b6cc942dc5b7c806f91b38164d26d96be988875d68f", + "activatedObjectId": "failure:48a0ab17763ecd1e3d09fbd6", + "eventIds": [ + "019febb3-19af-7591-b0a5-fa36870abe8c", + "019febb3-19b1-7a52-95df-b4ee618261f7", + "019febb3-19b2-72c8-b8ba-480e2cbd9340", + "019febb3-19b2-7d36-bac8-7007a6482160" + ], + "receiptHash": "7f2feee12030cf9e2f6715cd3e70e40e4673cbeb700c214e8fb7fd9bfb4a9e43" + }, + "workUsage": { + "inputTokens": 283538, + "cachedInputTokens": 270537, + "outputTokens": 2070, + "totalTokens": 285608, + "costUsd": 0.19137909999999997, + "tokenAccounting": "provider-cli-native-logical-input" + }, + "harvestUsage": { + "inputTokens": 0, + "outputTokens": 322, + "totalTokens": 322, + "modelCalls": 0, + "tokenAccounting": "model-output-delta:tiktoken-compatible-estimate" + }, + "usage": { + "inputTokens": 283538, + "cachedInputTokens": 270537, + "outputTokens": 2070, + "totalTokens": 285608, + "costUsd": 0.19137909999999997, + "tokenAccounting": "provider-cli-native-logical-input" + }, + "latencyMs": 36061, + "harvestHostLatencyMs": 302.5072999999975 + }, + "runtime": { + "exitCode": 0, + "correct": true, + "delivered": true, + "selected": true, + "retrievalAttempted": true, + "attested": false, + "attestationFailed": false, + "deliveryPhase": "adapter-pre-action", + "nativeGuardWired": true, + "nativeGuardEnforced": false, + "nativeGuardAuditEvents": 0, + "mistakeExecuted": false, + "outcomeGrade": { + "passed": true, + "protectedFilesUnchanged": true, + "canonicalSourceChanged": true, + "generatedSynchronized": true, + "verifierExitCode": 0, + "verifierOutput": "verification passed", + "sourceHash": "17f411dba7fbceba8f0083fd231b07e49e871688a4e80e2f6a3391c61bedf1b5", + "generatedHash": "d7e2a71156c9a9593402d8b4138dedd25f49e19f2da5388f58c9d8dec9ef9fa4", + "latencyMs": 138.1923999999999 + }, + "usage": { + "inputTokens": 88367, + "cachedInputTokens": 74240, + "outputTokens": 1191, + "totalTokens": null, + "costUsd": null, + "tokenAccounting": "provider-cli-native", + "staticSchemaTokens": 0, + "capsuleTokens": 98, + "instructionTokens": 119 + }, + "latencyMs": 41548, + "preflightLatencyMs": 163, + "firstSuccessorCost": { + "captureTokens": 322, + "additionalModelCalls": 0, + "hostCaptureLatencyMs": 302.5072999999975 + }, + "costLedger": { + "schemaVersion": "ucr.cognitive-cost-ledger/1", + "runId": "multimodel-claude-code-codex-b25920ecf84c60f3:runtime", + "entries": [ + { + "phase": "schema", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "consumer-tools-list:tiktoken", + "includedInTotal": true, + "detail": { + "exposedTools": 0 + } + }, + { + "phase": "capture", + "inputTokens": 283538, + "outputTokens": 2070, + "latencyMs": 36363.5073, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native-logical-input", + "includedInTotal": true, + "detail": { + "semanticDeltaTokens": 322, + "additionalModelCalls": 0 + } + }, + { + "phase": "retrieval", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 163, + "toolCalls": 1, + "modelCalls": 0, + "accountingMethod": "host-preaction-timer", + "includedInTotal": true, + "detail": null + }, + { + "phase": "injection", + "inputTokens": 119, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "trusted-context:utf8-length-div-4", + "includedInTotal": false, + "detail": { + "alreadyIncludedInConsumerInput": true, + "capsuleTokens": 98 + } + }, + { + "phase": "consumer", + "inputTokens": 88367, + "outputTokens": 1191, + "latencyMs": 41548, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native", + "includedInTotal": true, + "detail": null + }, + { + "phase": "validation", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 282.2514000000083, + "toolCalls": 2, + "modelCalls": 0, + "accountingMethod": "independent-grader-timer", + "includedInTotal": true, + "detail": null + } + ], + "totals": { + "inputTokens": 371905, + "outputTokens": 3261, + "totalTokens": 375166, + "latencyMs": 78356.7587, + "toolCalls": 3, + "modelCalls": 2 + }, + "missingPhases": [], + "attributionComplete": true, + "ledgerHash": "412b8024caf92f12433095d4875b78f4c0b5378cace1ac8b96b886cb14e151c4" + }, + "pipelineUsage": { + "inputTokens": 371905, + "outputTokens": 3261, + "totalTokens": 375166, + "latencyMs": 78356.7587, + "toolCalls": 3, + "modelCalls": 2, + "tokenAccounting": "phase-ledger:provider-native" + }, + "consumerMcpExposed": false, + "preActionReceipt": { + "schemaVersion": "ucr.pre-action-receipt/1", + "retrievalAttempted": true, + "action": "deliver", + "delivered": true, + "taskId": "multimodel-claude-code-codex-b25920ecf84c60f3", + "sessionId": "consumer-session", + "trigger": "task", + "budget": 160, + "capsuleIds": [ + "capsule:c7f9e4017778ccbd3aa49915" + ], + "objectIds": [ + "failure:48a0ab17763ecd1e3d09fbd6" + ], + "capsuleTokens": 98, + "injectionTokens": 119, + "staticSchemaTokens": 0, + "consumerMcpExposed": false, + "exposedTools": [], + "injectionChannel": "codex-developer-instructions", + "deliveryEventId": "019febb3-1a5b-7eb4-a134-0eca6cd147db", + "latencyMs": 163, + "receiptHash": "125fbdfb9bda2de29a698828edbb1dbfff61df8a825342a3f19127b7be107225" + } + }, + "eventStreamHash": "8fe978479f50d9c2cfd31215f812250d6a681dd7c24957ef82108894fc94d2c1", + "canonicalEvents": 5, + "passed": true, + "transcriptPublished": false, + "studySequence": null, + "schemaVersion": "ucr.evidence-row/2", + "runId": "live-multimodel-70ddcc01aef33de0", + "sequence": 0, + "previousHash": "4a5e0f813f2bb5a3237fafadfde21c949daf8b3fb76de2741de694b93751f7b2", + "rowHash": "09b3554fcc29819b77a9892f35ffb346e75923850cc30ea56dc82fe1ffbb7f66" + } + ], + "rowCount": 1, + "chainHead": "09b3554fcc29819b77a9892f35ffb346e75923850cc30ea56dc82fe1ffbb7f66", + "transcriptsPublished": false, + "endedAt": "2026-08-10T12:44:14.816Z", + "ledgerHash": "b8c8371750b77e12f60507be373aedbca433ecabf7c9a29ba3ecf842ad07822d", + "signature": "FKHcYghf+gs7cttKrOBxKcEEPj1d4uw3Dv+pr4N1InUBpbzHt90H00PkMXsO7IHLUno4DaaPh7aBJOrNaaYkCA==" + }, + "ledgerPublicKey": "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VwAyEAH8CygMdtyZVnbjcmAKQoREgOuZpkIrFlL3XNQOpF9jk=\n-----END PUBLIC KEY-----\n", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "b8c8371750b77e12f60507be373aedbca433ecabf7c9a29ba3ecf842ad07822d", + "rowCount": 1 + }, + "transcriptPublished": false, + "passed": true, + "limitations": [ + "one task per direction is executable coverage, not a powered effectiveness estimate", + "mandatory adapter preflight is evaluated here; advisory-only clients require separate evidence and cannot claim guaranteed prevention", + "provider aliases require provider response metadata before exact model-version claims", + "the installed standalone Gemini CLI and the advertised Copilot Gemini model were unavailable to the authenticated accounts; their failed attempts remain separate negative evidence", + "no competitive superiority claim is permitted from this smoke matrix" + ], + "reportHash": "0fe914b29aece48d91c10f6a23c7b09e02b79f298e01829cafe0c9ab729d241e" +} diff --git a/evals/ucr/results/stateful-matched-codex-to-claude-v1.json b/evals/ucr/results/stateful-matched-codex-to-claude-v1.json new file mode 100644 index 000000000..566c5bd5e --- /dev/null +++ b/evals/ucr/results/stateful-matched-codex-to-claude-v1.json @@ -0,0 +1,893 @@ +{ + "schemaVersion": "ucr.live-multimodel-handoff/1", + "evidenceClass": "live-multimodel-executable-smoke-not-powered-effectiveness", + "executedAt": "2026-08-10T12:39:01.830Z", + "sourceTreeHash": "96fcf325cd1945645579e83c48f0164c569017fdf952546ebe174f225c24df6d", + "requestedDirections": [ + "codex->claude-code" + ], + "directionsPassed": 1, + "directionResults": [ + { + "study": "cross-client-handoff", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "sonnet", + "taskHash": "b6b79053b282a7ea13364797f5c3c098f9c94e1eec8f6ba2847ef3002e3845cb", + "control": { + "exitCode": 0, + "correct": true, + "abstained": false, + "mistakeExecuted": false, + "outcomeGrade": { + "passed": true, + "protectedFilesUnchanged": true, + "canonicalSourceChanged": true, + "generatedSynchronized": true, + "verifierExitCode": 0, + "verifierOutput": "verification passed", + "sourceHash": "bfe31a9f5cb51d6096480a0b507e9ddab94db38ff6f40e06b62f49c32d5a835d", + "generatedHash": "bf047ff6229c44ecc59db82079f7a0bede2323c2668293c9bbcfb5a84ad3ae6e", + "latencyMs": 160.2514000000083 + }, + "usage": { + "inputTokens": 236744, + "cachedInputTokens": 224438, + "outputTokens": 1849, + "totalTokens": 238593, + "costUsd": 0.16960240000000001, + "tokenAccounting": "provider-cli-native-logical-input" + }, + "latencyMs": 26857, + "predecessorUsage": { + "inputTokens": 85261, + "cachedInputTokens": 77312, + "outputTokens": 1032, + "totalTokens": null, + "costUsd": null, + "tokenAccounting": "provider-cli-native" + }, + "predecessorLatencyMs": 40354, + "costLedger": { + "schemaVersion": "ucr.cognitive-cost-ledger/1", + "runId": "multimodel-codex-claude-code-7307ea21f2552640:control", + "entries": [ + { + "phase": "schema", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "consumer-tools-list:tiktoken", + "includedInTotal": true, + "detail": { + "exposedTools": 0 + } + }, + { + "phase": "capture", + "inputTokens": 85261, + "outputTokens": 1032, + "latencyMs": 40354, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native", + "includedInTotal": true, + "detail": { + "semanticDeltaTokens": 0, + "additionalModelCalls": 0 + } + }, + { + "phase": "retrieval", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "host-preaction-timer", + "includedInTotal": true, + "detail": null + }, + { + "phase": "injection", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "trusted-context:utf8-length-div-4", + "includedInTotal": false, + "detail": { + "alreadyIncludedInConsumerInput": true, + "capsuleTokens": 0 + } + }, + { + "phase": "consumer", + "inputTokens": 236744, + "outputTokens": 1849, + "latencyMs": 26857, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native-logical-input", + "includedInTotal": true, + "detail": null + }, + { + "phase": "validation", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 305.1478000000061, + "toolCalls": 2, + "modelCalls": 0, + "accountingMethod": "independent-grader-timer", + "includedInTotal": true, + "detail": null + } + ], + "totals": { + "inputTokens": 322005, + "outputTokens": 2881, + "totalTokens": 324886, + "latencyMs": 67516.1478, + "toolCalls": 2, + "modelCalls": 2 + }, + "missingPhases": [], + "attributionComplete": true, + "ledgerHash": "8a9435feb6bf342e75159033c611d77fa6672de6de33cd9a3c29492b5668f4ef" + }, + "pipelineUsage": { + "inputTokens": 322005, + "outputTokens": 2881, + "totalTokens": 324886, + "latencyMs": 67516.1478, + "toolCalls": 2, + "modelCalls": 2, + "tokenAccounting": "phase-ledger:provider-native" + } + }, + "producer": { + "exitCode": 0, + "recorded": true, + "modelAuthored": true, + "authoredDuringWorkTurn": true, + "evidenceAuthenticatedBeforeActivation": true, + "additionalModelCalls": 0, + "guardMaterialized": true, + "predecessorMistakeObserved": true, + "predecessorCorrectionVerified": true, + "predecessorEvidence": { + "trigger": "continuing implementation task multimodel-codex-claude-code-7307ea21f2552640", + "attemptedAction": "apply implementation choice EDIT-GENERATED-CLIENT-HOOK", + "observedFailure": "regeneration verifier rejected EDIT-GENERATED-CLIENT-HOOK after overwriting the direct generated-hook edit", + "rootCause": "client hook copies are generated from hooks-core and direct edits do not survive synchronization", + "correction": "apply verified implementation choice EDIT-HOOKS-CORE-THEN-SYNC", + "verificationEvidence": "external audit observed failed regeneration followed by a passing end-state grade for EDIT-HOOKS-CORE-THEN-SYNC", + "expectedOutcome": "task multimodel-codex-claude-code-7307ea21f2552640 updates hooks-core, regenerates clients, and does not repeat EDIT-GENERATED-CLIENT-HOOK", + "staleAttemptObserved": true, + "correctionObserved": true, + "correctedAttemptAccepted": true, + "auditHash": "f5dce5e6d60e3c280823d0e601d8764150fb6f793cf30dc644b88d74146f8559", + "finalGrade": { + "passed": true, + "protectedFilesUnchanged": true, + "canonicalSourceChanged": true, + "generatedSynchronized": true, + "verifierExitCode": 0, + "verifierOutput": "verification passed", + "sourceHash": "bfe31a9f5cb51d6096480a0b507e9ddab94db38ff6f40e06b62f49c32d5a835d", + "generatedHash": "bf047ff6229c44ecc59db82079f7a0bede2323c2668293c9bbcfb5a84ad3ae6e", + "latencyMs": 155.4212000000116 + } + }, + "semanticHarvestReceipt": { + "schemaVersion": "ucr.semantic-harvest-receipt/2", + "modelAuthored": true, + "authoredDuringWorkTurn": true, + "evidenceAuthenticatedBeforeActivation": true, + "additionalModelCalls": 0, + "kind": "failure", + "taskId": "multimodel-codex-claude-code-7307ea21f2552640", + "deltaTokens": 267, + "hostLatencyMs": 283.8990000000049, + "modelOutputHash": "ccb83805d470c4653615bbb8c68cebaae8d10db85aca7aec7a4c1ace446ec25b", + "semanticObjectHash": "83b3d7741f9bdf7ac012e4ec3035907e50914d22da67c078528b133b89791600", + "activatedObjectId": "failure:9182fa04929ed0b7f818dcc0", + "eventIds": [ + "019febae-76b3-7ab5-997d-b54b535df7aa", + "019febae-76b4-7ab4-a2b2-7f18eb07981c", + "019febae-76b5-75fb-8109-8f7ad3ef392b", + "019febae-76b5-7c1d-bb52-86ee783ad60c" + ], + "receiptHash": "9ebf42b6a45b8cec374172411239fecaf5464e5337e843bfd0ae747f2611132a" + }, + "workUsage": { + "inputTokens": 85607, + "cachedInputTokens": 74240, + "outputTokens": 1289, + "totalTokens": null, + "costUsd": null, + "tokenAccounting": "provider-cli-native" + }, + "harvestUsage": { + "inputTokens": 0, + "outputTokens": 267, + "totalTokens": 267, + "modelCalls": 0, + "tokenAccounting": "model-output-delta:tiktoken-compatible-estimate" + }, + "usage": { + "inputTokens": 85607, + "cachedInputTokens": 74240, + "outputTokens": 1289, + "totalTokens": null, + "costUsd": null, + "tokenAccounting": "provider-cli-native" + }, + "latencyMs": 45493, + "harvestHostLatencyMs": 283.8990000000049 + }, + "runtime": { + "exitCode": 0, + "correct": true, + "delivered": true, + "selected": true, + "retrievalAttempted": true, + "attested": false, + "attestationFailed": false, + "deliveryPhase": "adapter-pre-action", + "nativeGuardWired": true, + "nativeGuardEnforced": false, + "nativeGuardAuditEvents": 0, + "mistakeExecuted": false, + "outcomeGrade": { + "passed": true, + "protectedFilesUnchanged": true, + "canonicalSourceChanged": true, + "generatedSynchronized": true, + "verifierExitCode": 0, + "verifierOutput": "verification passed", + "sourceHash": "bfe31a9f5cb51d6096480a0b507e9ddab94db38ff6f40e06b62f49c32d5a835d", + "generatedHash": "bf047ff6229c44ecc59db82079f7a0bede2323c2668293c9bbcfb5a84ad3ae6e", + "latencyMs": 158.93359999998938 + }, + "usage": { + "inputTokens": 198736, + "cachedInputTokens": 186401, + "outputTokens": 1712, + "totalTokens": 200448, + "costUsd": 0.15630629999999998, + "tokenAccounting": "provider-cli-native-logical-input", + "staticSchemaTokens": 0, + "capsuleTokens": 57, + "instructionTokens": 77 + }, + "latencyMs": 28617, + "preflightLatencyMs": 145, + "firstSuccessorCost": { + "captureTokens": 267, + "additionalModelCalls": 0, + "hostCaptureLatencyMs": 283.8990000000049 + }, + "costLedger": { + "schemaVersion": "ucr.cognitive-cost-ledger/1", + "runId": "multimodel-codex-claude-code-7307ea21f2552640:runtime", + "entries": [ + { + "phase": "schema", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "consumer-tools-list:tiktoken", + "includedInTotal": true, + "detail": { + "exposedTools": 0 + } + }, + { + "phase": "capture", + "inputTokens": 85607, + "outputTokens": 1289, + "latencyMs": 45776.899000000005, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native", + "includedInTotal": true, + "detail": { + "semanticDeltaTokens": 267, + "additionalModelCalls": 0 + } + }, + { + "phase": "retrieval", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 145, + "toolCalls": 1, + "modelCalls": 0, + "accountingMethod": "host-preaction-timer", + "includedInTotal": true, + "detail": null + }, + { + "phase": "injection", + "inputTokens": 77, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "trusted-context:utf8-length-div-4", + "includedInTotal": false, + "detail": { + "alreadyIncludedInConsumerInput": true, + "capsuleTokens": 57 + } + }, + { + "phase": "consumer", + "inputTokens": 198736, + "outputTokens": 1712, + "latencyMs": 28617, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native-logical-input", + "includedInTotal": true, + "detail": null + }, + { + "phase": "validation", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 314.354800000001, + "toolCalls": 2, + "modelCalls": 0, + "accountingMethod": "independent-grader-timer", + "includedInTotal": true, + "detail": null + } + ], + "totals": { + "inputTokens": 284343, + "outputTokens": 3001, + "totalTokens": 287344, + "latencyMs": 74853.2538, + "toolCalls": 3, + "modelCalls": 2 + }, + "missingPhases": [], + "attributionComplete": true, + "ledgerHash": "30ef63671a07386e377f84ee6756867ce551d980192dc7de76cef22588ff966b" + }, + "pipelineUsage": { + "inputTokens": 284343, + "outputTokens": 3001, + "totalTokens": 287344, + "latencyMs": 74853.2538, + "toolCalls": 3, + "modelCalls": 2, + "tokenAccounting": "phase-ledger:provider-native" + }, + "consumerMcpExposed": false, + "preActionReceipt": { + "schemaVersion": "ucr.pre-action-receipt/1", + "retrievalAttempted": true, + "action": "deliver", + "delivered": true, + "taskId": "multimodel-codex-claude-code-7307ea21f2552640", + "sessionId": "consumer-session", + "trigger": "task", + "budget": 160, + "capsuleIds": [ + "capsule:ba496649483b6b8b6290e61b" + ], + "objectIds": [ + "failure:9182fa04929ed0b7f818dcc0" + ], + "capsuleTokens": 57, + "injectionTokens": 77, + "staticSchemaTokens": 0, + "consumerMcpExposed": false, + "exposedTools": [], + "injectionChannel": "claude-host-appended-system-prompt", + "deliveryEventId": "019febae-774f-794e-99a3-781d17e90a05", + "latencyMs": 145, + "receiptHash": "c2aa57fdafc2bbaa62717924465589d58823830aa0e3d5a7a365fb0b1ea5bcd8" + } + }, + "eventStreamHash": "3fd304480a3ade88b0bc806231d3115943f63b1ebacfd28ba19bb4e3da048f71", + "canonicalEvents": 5, + "passed": true, + "transcriptPublished": false + } + ], + "clients": { + "codex": { + "family": "openai", + "model": "gpt-5.6-sol", + "version": "codex-cli 0.147.0" + }, + "claude-code": { + "family": "anthropic", + "model": "sonnet", + "version": "2.1.225 (Claude Code)" + }, + "copilot": { + "family": "anthropic", + "model": "claude-sonnet-4.5", + "version": "GitHub Copilot CLI 0.0.396.\nRun 'copilot update' to check for updates." + } + }, + "modelFamilies": 2, + "attestationSurface": { + "profile": "attestation", + "toolNames": [ + "context_receipt_verify" + ], + "schemaTokens": 71, + "tokenAccounting": "tiktoken:cl100k_base" + }, + "controlsAbstained": 0, + "repeatedVerifiedFailures": 0, + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "live-multimodel-6389fc59c3f37bff", + "evidenceClass": "executable-smoke", + "benchmarkHash": "4b2148b8a73d20f9d94a3008e1756f9fb38144818bb527b752ec5ff6ffa06f0e", + "sourceTreeHash": "96fcf325cd1945645579e83c48f0164c569017fdf952546ebe174f225c24df6d", + "runner": { + "name": "live-multimodel-handoff", + "versions": { + "codex": "codex-cli 0.147.0", + "claude-code": "2.1.225 (Claude Code)", + "copilot": "GitHub Copilot CLI 0.0.396.\nRun 'copilot update' to check for updates." + } + }, + "startedAt": "2026-08-10T12:39:01.823Z", + "runHash": "71cabfac08159db727f0896239b18aa4fed032be40629b25b7093e2bc8a95c4d" + }, + "rows": [ + { + "study": "cross-client-handoff", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "sonnet", + "taskHash": "b6b79053b282a7ea13364797f5c3c098f9c94e1eec8f6ba2847ef3002e3845cb", + "control": { + "exitCode": 0, + "correct": true, + "abstained": false, + "mistakeExecuted": false, + "outcomeGrade": { + "passed": true, + "protectedFilesUnchanged": true, + "canonicalSourceChanged": true, + "generatedSynchronized": true, + "verifierExitCode": 0, + "verifierOutput": "verification passed", + "sourceHash": "bfe31a9f5cb51d6096480a0b507e9ddab94db38ff6f40e06b62f49c32d5a835d", + "generatedHash": "bf047ff6229c44ecc59db82079f7a0bede2323c2668293c9bbcfb5a84ad3ae6e", + "latencyMs": 160.2514000000083 + }, + "usage": { + "inputTokens": 236744, + "cachedInputTokens": 224438, + "outputTokens": 1849, + "totalTokens": 238593, + "costUsd": 0.16960240000000001, + "tokenAccounting": "provider-cli-native-logical-input" + }, + "latencyMs": 26857, + "predecessorUsage": { + "inputTokens": 85261, + "cachedInputTokens": 77312, + "outputTokens": 1032, + "totalTokens": null, + "costUsd": null, + "tokenAccounting": "provider-cli-native" + }, + "predecessorLatencyMs": 40354, + "costLedger": { + "schemaVersion": "ucr.cognitive-cost-ledger/1", + "runId": "multimodel-codex-claude-code-7307ea21f2552640:control", + "entries": [ + { + "phase": "schema", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "consumer-tools-list:tiktoken", + "includedInTotal": true, + "detail": { + "exposedTools": 0 + } + }, + { + "phase": "capture", + "inputTokens": 85261, + "outputTokens": 1032, + "latencyMs": 40354, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native", + "includedInTotal": true, + "detail": { + "semanticDeltaTokens": 0, + "additionalModelCalls": 0 + } + }, + { + "phase": "retrieval", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "host-preaction-timer", + "includedInTotal": true, + "detail": null + }, + { + "phase": "injection", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "trusted-context:utf8-length-div-4", + "includedInTotal": false, + "detail": { + "alreadyIncludedInConsumerInput": true, + "capsuleTokens": 0 + } + }, + { + "phase": "consumer", + "inputTokens": 236744, + "outputTokens": 1849, + "latencyMs": 26857, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native-logical-input", + "includedInTotal": true, + "detail": null + }, + { + "phase": "validation", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 305.1478000000061, + "toolCalls": 2, + "modelCalls": 0, + "accountingMethod": "independent-grader-timer", + "includedInTotal": true, + "detail": null + } + ], + "totals": { + "inputTokens": 322005, + "outputTokens": 2881, + "totalTokens": 324886, + "latencyMs": 67516.1478, + "toolCalls": 2, + "modelCalls": 2 + }, + "missingPhases": [], + "attributionComplete": true, + "ledgerHash": "8a9435feb6bf342e75159033c611d77fa6672de6de33cd9a3c29492b5668f4ef" + }, + "pipelineUsage": { + "inputTokens": 322005, + "outputTokens": 2881, + "totalTokens": 324886, + "latencyMs": 67516.1478, + "toolCalls": 2, + "modelCalls": 2, + "tokenAccounting": "phase-ledger:provider-native" + } + }, + "producer": { + "exitCode": 0, + "recorded": true, + "modelAuthored": true, + "authoredDuringWorkTurn": true, + "evidenceAuthenticatedBeforeActivation": true, + "additionalModelCalls": 0, + "guardMaterialized": true, + "predecessorMistakeObserved": true, + "predecessorCorrectionVerified": true, + "predecessorEvidence": { + "trigger": "continuing implementation task multimodel-codex-claude-code-7307ea21f2552640", + "attemptedAction": "apply implementation choice EDIT-GENERATED-CLIENT-HOOK", + "observedFailure": "regeneration verifier rejected EDIT-GENERATED-CLIENT-HOOK after overwriting the direct generated-hook edit", + "rootCause": "client hook copies are generated from hooks-core and direct edits do not survive synchronization", + "correction": "apply verified implementation choice EDIT-HOOKS-CORE-THEN-SYNC", + "verificationEvidence": "external audit observed failed regeneration followed by a passing end-state grade for EDIT-HOOKS-CORE-THEN-SYNC", + "expectedOutcome": "task multimodel-codex-claude-code-7307ea21f2552640 updates hooks-core, regenerates clients, and does not repeat EDIT-GENERATED-CLIENT-HOOK", + "staleAttemptObserved": true, + "correctionObserved": true, + "correctedAttemptAccepted": true, + "auditHash": "f5dce5e6d60e3c280823d0e601d8764150fb6f793cf30dc644b88d74146f8559", + "finalGrade": { + "passed": true, + "protectedFilesUnchanged": true, + "canonicalSourceChanged": true, + "generatedSynchronized": true, + "verifierExitCode": 0, + "verifierOutput": "verification passed", + "sourceHash": "bfe31a9f5cb51d6096480a0b507e9ddab94db38ff6f40e06b62f49c32d5a835d", + "generatedHash": "bf047ff6229c44ecc59db82079f7a0bede2323c2668293c9bbcfb5a84ad3ae6e", + "latencyMs": 155.4212000000116 + } + }, + "semanticHarvestReceipt": { + "schemaVersion": "ucr.semantic-harvest-receipt/2", + "modelAuthored": true, + "authoredDuringWorkTurn": true, + "evidenceAuthenticatedBeforeActivation": true, + "additionalModelCalls": 0, + "kind": "failure", + "taskId": "multimodel-codex-claude-code-7307ea21f2552640", + "deltaTokens": 267, + "hostLatencyMs": 283.8990000000049, + "modelOutputHash": "ccb83805d470c4653615bbb8c68cebaae8d10db85aca7aec7a4c1ace446ec25b", + "semanticObjectHash": "83b3d7741f9bdf7ac012e4ec3035907e50914d22da67c078528b133b89791600", + "activatedObjectId": "failure:9182fa04929ed0b7f818dcc0", + "eventIds": [ + "019febae-76b3-7ab5-997d-b54b535df7aa", + "019febae-76b4-7ab4-a2b2-7f18eb07981c", + "019febae-76b5-75fb-8109-8f7ad3ef392b", + "019febae-76b5-7c1d-bb52-86ee783ad60c" + ], + "receiptHash": "9ebf42b6a45b8cec374172411239fecaf5464e5337e843bfd0ae747f2611132a" + }, + "workUsage": { + "inputTokens": 85607, + "cachedInputTokens": 74240, + "outputTokens": 1289, + "totalTokens": null, + "costUsd": null, + "tokenAccounting": "provider-cli-native" + }, + "harvestUsage": { + "inputTokens": 0, + "outputTokens": 267, + "totalTokens": 267, + "modelCalls": 0, + "tokenAccounting": "model-output-delta:tiktoken-compatible-estimate" + }, + "usage": { + "inputTokens": 85607, + "cachedInputTokens": 74240, + "outputTokens": 1289, + "totalTokens": null, + "costUsd": null, + "tokenAccounting": "provider-cli-native" + }, + "latencyMs": 45493, + "harvestHostLatencyMs": 283.8990000000049 + }, + "runtime": { + "exitCode": 0, + "correct": true, + "delivered": true, + "selected": true, + "retrievalAttempted": true, + "attested": false, + "attestationFailed": false, + "deliveryPhase": "adapter-pre-action", + "nativeGuardWired": true, + "nativeGuardEnforced": false, + "nativeGuardAuditEvents": 0, + "mistakeExecuted": false, + "outcomeGrade": { + "passed": true, + "protectedFilesUnchanged": true, + "canonicalSourceChanged": true, + "generatedSynchronized": true, + "verifierExitCode": 0, + "verifierOutput": "verification passed", + "sourceHash": "bfe31a9f5cb51d6096480a0b507e9ddab94db38ff6f40e06b62f49c32d5a835d", + "generatedHash": "bf047ff6229c44ecc59db82079f7a0bede2323c2668293c9bbcfb5a84ad3ae6e", + "latencyMs": 158.93359999998938 + }, + "usage": { + "inputTokens": 198736, + "cachedInputTokens": 186401, + "outputTokens": 1712, + "totalTokens": 200448, + "costUsd": 0.15630629999999998, + "tokenAccounting": "provider-cli-native-logical-input", + "staticSchemaTokens": 0, + "capsuleTokens": 57, + "instructionTokens": 77 + }, + "latencyMs": 28617, + "preflightLatencyMs": 145, + "firstSuccessorCost": { + "captureTokens": 267, + "additionalModelCalls": 0, + "hostCaptureLatencyMs": 283.8990000000049 + }, + "costLedger": { + "schemaVersion": "ucr.cognitive-cost-ledger/1", + "runId": "multimodel-codex-claude-code-7307ea21f2552640:runtime", + "entries": [ + { + "phase": "schema", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "consumer-tools-list:tiktoken", + "includedInTotal": true, + "detail": { + "exposedTools": 0 + } + }, + { + "phase": "capture", + "inputTokens": 85607, + "outputTokens": 1289, + "latencyMs": 45776.899000000005, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native", + "includedInTotal": true, + "detail": { + "semanticDeltaTokens": 267, + "additionalModelCalls": 0 + } + }, + { + "phase": "retrieval", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 145, + "toolCalls": 1, + "modelCalls": 0, + "accountingMethod": "host-preaction-timer", + "includedInTotal": true, + "detail": null + }, + { + "phase": "injection", + "inputTokens": 77, + "outputTokens": 0, + "latencyMs": 0, + "toolCalls": 0, + "modelCalls": 0, + "accountingMethod": "trusted-context:utf8-length-div-4", + "includedInTotal": false, + "detail": { + "alreadyIncludedInConsumerInput": true, + "capsuleTokens": 57 + } + }, + { + "phase": "consumer", + "inputTokens": 198736, + "outputTokens": 1712, + "latencyMs": 28617, + "toolCalls": 0, + "modelCalls": 1, + "accountingMethod": "provider-cli-native-logical-input", + "includedInTotal": true, + "detail": null + }, + { + "phase": "validation", + "inputTokens": 0, + "outputTokens": 0, + "latencyMs": 314.354800000001, + "toolCalls": 2, + "modelCalls": 0, + "accountingMethod": "independent-grader-timer", + "includedInTotal": true, + "detail": null + } + ], + "totals": { + "inputTokens": 284343, + "outputTokens": 3001, + "totalTokens": 287344, + "latencyMs": 74853.2538, + "toolCalls": 3, + "modelCalls": 2 + }, + "missingPhases": [], + "attributionComplete": true, + "ledgerHash": "30ef63671a07386e377f84ee6756867ce551d980192dc7de76cef22588ff966b" + }, + "pipelineUsage": { + "inputTokens": 284343, + "outputTokens": 3001, + "totalTokens": 287344, + "latencyMs": 74853.2538, + "toolCalls": 3, + "modelCalls": 2, + "tokenAccounting": "phase-ledger:provider-native" + }, + "consumerMcpExposed": false, + "preActionReceipt": { + "schemaVersion": "ucr.pre-action-receipt/1", + "retrievalAttempted": true, + "action": "deliver", + "delivered": true, + "taskId": "multimodel-codex-claude-code-7307ea21f2552640", + "sessionId": "consumer-session", + "trigger": "task", + "budget": 160, + "capsuleIds": [ + "capsule:ba496649483b6b8b6290e61b" + ], + "objectIds": [ + "failure:9182fa04929ed0b7f818dcc0" + ], + "capsuleTokens": 57, + "injectionTokens": 77, + "staticSchemaTokens": 0, + "consumerMcpExposed": false, + "exposedTools": [], + "injectionChannel": "claude-host-appended-system-prompt", + "deliveryEventId": "019febae-774f-794e-99a3-781d17e90a05", + "latencyMs": 145, + "receiptHash": "c2aa57fdafc2bbaa62717924465589d58823830aa0e3d5a7a365fb0b1ea5bcd8" + } + }, + "eventStreamHash": "3fd304480a3ade88b0bc806231d3115943f63b1ebacfd28ba19bb4e3da048f71", + "canonicalEvents": 5, + "passed": true, + "transcriptPublished": false, + "studySequence": null, + "schemaVersion": "ucr.evidence-row/2", + "runId": "live-multimodel-6389fc59c3f37bff", + "sequence": 0, + "previousHash": "71cabfac08159db727f0896239b18aa4fed032be40629b25b7093e2bc8a95c4d", + "rowHash": "2abb57c6f6531438f67bc6db165cefede1d457d2d3e25c2b8848b26f6aba865f" + } + ], + "rowCount": 1, + "chainHead": "2abb57c6f6531438f67bc6db165cefede1d457d2d3e25c2b8848b26f6aba865f", + "transcriptsPublished": false, + "endedAt": "2026-08-10T12:39:01.825Z", + "ledgerHash": "0efd8b4dcaa975017b0509c47906b34d7b836d45f3d3e976be787fd4ea2a2c41", + "signature": "mXYp/1+kC6ICr+ng0eYvv+1MvXmCguj/HwCZq00qE/MwdbjoMXFu+dgaICh4M4njWSBwi0OH4FQXNmO4a1JcDw==" + }, + "ledgerPublicKey": "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VwAyEA9BaxtPZAngiQxhg+lK4/98QL48Xk8u4yGWOWCi6fbm8=\n-----END PUBLIC KEY-----\n", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "0efd8b4dcaa975017b0509c47906b34d7b836d45f3d3e976be787fd4ea2a2c41", + "rowCount": 1 + }, + "transcriptPublished": false, + "passed": true, + "limitations": [ + "one task per direction is executable coverage, not a powered effectiveness estimate", + "mandatory adapter preflight is evaluated here; advisory-only clients require separate evidence and cannot claim guaranteed prevention", + "provider aliases require provider response metadata before exact model-version claims", + "the installed standalone Gemini CLI and the advertised Copilot Gemini model were unavailable to the authenticated accounts; their failed attempts remain separate negative evidence", + "no competitive superiority claim is permitted from this smoke matrix" + ], + "reportHash": "7b2b74545eed9ef9eb78dbc9dab6ba23f7e56a1cf121473830e4a3a28ad9695e" +} diff --git a/evals/ucr/results/study-readiness-v1.json b/evals/ucr/results/study-readiness-v1.json new file mode 100644 index 000000000..a99e35d01 --- /dev/null +++ b/evals/ucr/results/study-readiness-v1.json @@ -0,0 +1,56 @@ +{ + "schemaVersion": "ucr.study-readiness/1", + "passed": true, + "structuralPlanHash": "45d9dd4e582bbb0b1961e2084518a7bac26a45b95bf66c5360f18b4973263270", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "pairedRepetitionsPerCell": 33, + "primaryPairsPerDirection": 363, + "negativeRepetitionsPerCell": 96, + "negativeSamplesPerDirectionPerArm": 1056, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "trials": 54054, + "providerInvocations": 113022, + "mappedMetrics": 37, + "representativeStudyClients": 3, + "universalDriverClients": 16, + "driverContractPassed": true, + "missingMetrics": [], + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 1420 + }, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "reportHash": "02488e8c1f6d1b4495bdc14027778946ff69894e7709808b25d89f3e4235ace1" +} diff --git a/evals/ucr/results/superseded/full-study-blocked-claude-to-codex-v4-attempts.jsonl b/evals/ucr/results/superseded/full-study-blocked-claude-to-codex-v4-attempts.jsonl new file mode 100644 index 000000000..f641cafa0 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-blocked-claude-to-codex-v4-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":["executed model version differs from the frozen plan"],"providerFailures":[],"rowHash":"addcefb61ee95407d642ea84b82aa86a5838bee3bc40f28362de68fd172c3f82","runnerExitCode":0,"trialId":"claude-code->codex:federation-001:0:empty","valid":false,"validation":[]} +{"arm":"runtime","causalEvents":6,"driverValidation":["executed model version differs from the frozen plan"],"providerFailures":[],"rowHash":"81586721e394e424aa17094c3a4273a29b2602f105e7f11a21312aaf85fff3e8","runnerExitCode":0,"trialId":"claude-code->codex:federation-001:0:runtime","valid":false,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-blocked-claude-to-codex-v4.json b/evals/ucr/results/superseded/full-study-blocked-claude-to-codex-v4.json new file mode 100644 index 000000000..87dee7c76 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-blocked-claude-to-codex-v4.json @@ -0,0 +1,698 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T16:15:21.781Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "gemini-cli 0.28.1", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": false, + "crossSession": true, + "crossProject": true, + "concurrentAgents": false, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.2857142857142857, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 2, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "claude-code->codex", + "families": null, + "arms": "empty,runtime", + "requestedArms": "empty,runtime", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "97b8d742145106766dcbddae8271634e5945a8553263d5bb124f6b2110eb7318" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": false, + "trialIntegrity": false, + "taskCorrect": true, + "runtimeDelivered": true, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": true + }, + "failed": [ + "selectionComplete", + "trialIntegrity" + ], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": -0.4835535563677256 + }, + "failures": [ + { + "trialId": "claude-code->codex:federation-001:0:empty", + "arm": "empty", + "runnerExitCode": 0, + "valid": false, + "validation": [], + "driverValidation": [ + "executed model version differs from the frozen plan" + ], + "causalEvents": 0, + "providerFailures": [], + "rowHash": "addcefb61ee95407d642ea84b82aa86a5838bee3bc40f28362de68fd172c3f82" + }, + { + "trialId": "claude-code->codex:federation-001:0:runtime", + "arm": "runtime", + "runnerExitCode": 0, + "valid": false, + "validation": [], + "driverValidation": [ + "executed model version differs from the frozen plan" + ], + "causalEvents": 6, + "providerFailures": [], + "rowHash": "81586721e394e424aa17094c3a4273a29b2602f105e7f11a21312aaf85fff3e8" + } + ], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-ce253d11fe4fc373", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "b9eec6eae3d1a7f92fae68a8bec9ec3cccad7e4617895626c5e23101a250b0d4", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T16:15:21.775Z", + "runHash": "4b1c0d62136fb12adc05eded64db2bb6e22dcfd80536fa23caa63b1950a61bd5" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "claude-code->codex:federation-001:0:empty", + "pairId": "claude-code->codex:federation-001:0", + "studySequence": 52, + "taskId": "federation-001", + "family": "cross-project-generalization", + "arm": "empty", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "model": "gpt-5.6-sol", + "modelVersion": null, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "cross-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "99f2ebb96e250c908ad7449f409a0ff35e1c6d3c5c1f853b26108fea1426c297", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 86454, + "totalTokens": 96039, + "latencyMs": 128589, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "12724dc7e404c7c9e439b1d4a064736cb82b253023be78a6eea82fe933f56c51", + "workspaceStateHash": "eb156d253c3caeb43ed5cb056cf22629bb9da46869b1581a151d47f137672e89", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "309f9f1e0fd639c70cc7778fad659864ba17c381897adf3da914a7978eca3854", + "runnerExitCode": 0, + "runnerOutputHash": "5b0fdc34ae5dde62fef78addea70640bdc7ffe7d3cbeabde0908a6fd0f223c90", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-ce253d11fe4fc373", + "sequence": 0, + "previousHash": "4b1c0d62136fb12adc05eded64db2bb6e22dcfd80536fa23caa63b1950a61bd5", + "rowHash": "ca68627e5e3220d8e1d8772cd2c121d32c0a0d41a925cde2816276ae00c1b5d3" + }, + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "claude-code->codex:federation-001:0:runtime", + "pairId": "claude-code->codex:federation-001:0", + "studySequence": 551, + "taskId": "federation-001", + "family": "cross-project-generalization", + "arm": "runtime", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "model": "gpt-5.6-sol", + "modelVersion": null, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "cross-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "52c8a245c63f3a789640257921352cbcb66301faf0e652c34b2d35beedab82c0", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": true, + "deliveryPhase": "pre-action", + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0.008808921375691125, + "reconstructionTokens": 42684, + "totalTokens": 49599, + "latencyMs": 91810, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 376, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "7add49a3d67f4ced9974e2c833e9a5c69f164edd09ed32680edf0c35263f62e6", + "workspaceStateHash": "44acf7a5eb722e92c2be6cea5d287441e2180817c5e829e4677f3a6fb46a3555", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "aaea58899cfbc7b0336470990f1a3bccfe4be62e0cd30038a591a364314c532a", + "runnerExitCode": 0, + "runnerOutputHash": "47af2a79c923bf1e513c85f202f0434bb6fded467a4b52ef7a9b0078fd34b24b", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-ce253d11fe4fc373", + "sequence": 1, + "previousHash": "ca68627e5e3220d8e1d8772cd2c121d32c0a0d41a925cde2816276ae00c1b5d3", + "rowHash": "74f7621a367948a699d30074b17309079172dd4471d9f0e292328738e615e35d" + } + ], + "rowCount": 2, + "chainHead": "74f7621a367948a699d30074b17309079172dd4471d9f0e292328738e615e35d", + "transcriptsPublished": false, + "endedAt": "2026-08-10T16:15:21.776Z", + "ledgerHash": "e12971d09465588891f93f8c859eaf08ef41f4515aa531be50e2888a55ca8eaa", + "signature": "e8nVVG3T2DvFTA7ctYbfwGvEpDApH3eFskhMdeglNcXrlQaGhpoRkeAtR24rhry2pJqaTZgCG+Dv/ucCIi4+CQ==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "e12971d09465588891f93f8c859eaf08ef41f4515aa531be50e2888a55ca8eaa", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "e12971d09465588891f93f8c859eaf08ef41f4515aa531be50e2888a55ca8eaa" + ], + "rejectedLedgers": 0, + "derivedHash": "a01bd5f431b60f132dcf9e2f68d2b2dfc44a05580ff1acaf8811086928e22dcc" + }, + "costDiagnostics": { + "passed": true, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [ + { + "dimension": "direction", + "value": "claude-code->codex", + "pairs": 1, + "tokenOverhead": -0.4835535563677256, + "latencyOverheadP95": -0.2860197995162883, + "regressions": [], + "passed": true + }, + { + "dimension": "family", + "value": "cross-project-generalization", + "pairs": 1, + "tokenOverhead": -0.4835535563677256, + "latencyOverheadP95": -0.2860197995162883, + "regressions": [], + "passed": true + }, + { + "dimension": "client", + "value": "codex", + "pairs": 1, + "tokenOverhead": -0.4835535563677256, + "latencyOverheadP95": -0.2860197995162883, + "regressions": [], + "passed": true + } + ] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "cf3d8724e97d794cdbac8fe1a91642061dd213fd2412a4e64163be54b0027a54" +} diff --git a/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v1-attempts.jsonl b/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v1-attempts.jsonl new file mode 100644 index 000000000..49f3ad0a0 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v1-attempts.jsonl @@ -0,0 +1 @@ +{"arm":"empty","causalEvents":0,"driverValidation":["host action audit is incomplete","provider-native total token accounting is missing","executed CLI versions differ from the frozen plan","executed model version differs from the frozen plan","per-invocation phase telemetry is incomplete"],"providerFailures":[{"classification":"provider-client-migration-required","role":"consumer-1"}],"rowHash":"a26267f7ddc27a10ab3336fb45ecd544cae6d34c5d940612a708a7cfe829058f","runnerExitCode":1,"trialId":"codex->gemini:adversarial-001:0:empty","valid":false,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v1.json b/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v1.json new file mode 100644 index 000000000..115ed4b79 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v1.json @@ -0,0 +1,599 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T16:19:45.880Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "gemini-cli 0.28.1", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": true, + "crossSession": false, + "crossProject": false, + "concurrentAgents": false, + "pairedControls": false, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.14285714285714285, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 1, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 1, + "selection": { + "directions": "codex->gemini", + "families": null, + "arms": "empty", + "requestedArms": "empty", + "maxTrials": 1, + "selectedTrials": 1, + "selectedTrialIdsHash": "5dfb28fc49162ba20355c69e75cec0b1a1341507c09a4169b41e82ea3b79f5e8" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": false, + "trialIntegrity": false, + "taskCorrect": true, + "runtimeDelivered": true, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": true + }, + "failed": [ + "selectionComplete", + "trialIntegrity" + ], + "selectedTrials": 1, + "executedTrials": 1, + "completePairs": 0, + "maximumObservedTokenOverhead": null + }, + "failures": [ + { + "trialId": "codex->gemini:adversarial-001:0:empty", + "arm": "empty", + "runnerExitCode": 1, + "valid": false, + "validation": [], + "driverValidation": [ + "host action audit is incomplete", + "provider-native total token accounting is missing", + "executed CLI versions differ from the frozen plan", + "executed model version differs from the frozen plan", + "per-invocation phase telemetry is incomplete" + ], + "causalEvents": 0, + "providerFailures": [ + { + "classification": "provider-client-migration-required", + "role": "consumer-1" + } + ], + "rowHash": "a26267f7ddc27a10ab3336fb45ecd544cae6d34c5d940612a708a7cfe829058f" + } + ], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-a7807cd0826ff8a0", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "b9eec6eae3d1a7f92fae68a8bec9ec3cccad7e4617895626c5e23101a250b0d4", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T16:19:45.875Z", + "runHash": "3943e306599496834701f304902dd14ff5bc1eaaf5151edf2927dc8f9bbcf7d1" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "codex->gemini:adversarial-001:0:empty", + "pairId": "codex->gemini:adversarial-001:0", + "studySequence": 23, + "taskId": "adversarial-001", + "family": "adversarial-memory", + "arm": "empty", + "direction": "codex->gemini", + "producerClient": "codex", + "consumerClient": "gemini", + "producerFamily": "openai", + "consumerFamily": "google", + "producerModel": "gpt-5.6-sol", + "consumerModel": "gemini-2.5-pro", + "model": "gemini-2.5-pro", + "modelVersion": null, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "same-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "0b944e194f2c7261be0dfb240e752e3cfb41c95830299cb7b0f015ecffe7e080", + "poweredStratum": false, + "correct": false, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": null, + "totalTokens": null, + "latencyMs": 45345, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [ + { + "classification": "provider-client-migration-required", + "role": "consumer-1" + } + ], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "workspaceStateHash": "6f57f10a472583c0e8ed3ae52e682e812c8a47f6f6e9b10ed309307938c0948a", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "a0c638aca09379cebcd2984291ba99eb6a83f651597638019b9660bb4a14a221", + "runnerExitCode": 1, + "runnerOutputHash": "a50a07de3d366c1589d6556e4e149d7d473ea0eba795e47383df36464403c64b", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-a7807cd0826ff8a0", + "sequence": 0, + "previousHash": "3943e306599496834701f304902dd14ff5bc1eaaf5151edf2927dc8f9bbcf7d1", + "rowHash": "cbbde1a76ccbdacbd68aeaffd7c9aa8ef9075993df0515e4ff685d5432425750" + } + ], + "rowCount": 1, + "chainHead": "cbbde1a76ccbdacbd68aeaffd7c9aa8ef9075993df0515e4ff685d5432425750", + "transcriptsPublished": false, + "endedAt": "2026-08-10T16:19:45.876Z", + "ledgerHash": "1b259bd70702aa27b41861c2f721debeaeedd874f1435351f3b1af4ad4b59188", + "signature": "xdoyHhvl+NuIMRJQGKONsWGfTXsGu1CU+4jed/vmJuWOW+g9fXJpC6a7RF4tSYHiI+aQD1rJFzpf3hsmHLcSBQ==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "1b259bd70702aa27b41861c2f721debeaeedd874f1435351f3b1af4ad4b59188", + "rowCount": 1 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "1b259bd70702aa27b41861c2f721debeaeedd874f1435351f3b1af4ad4b59188" + ], + "rejectedLedgers": 0, + "derivedHash": "a614d22d2d7ef9407f022ace66ec7fc5b5620afd5a6eaa07ccfb7fd2e91aa7f3" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "be2097b18a46e28c9e916e69ae9a8914d6680e17e5f7aaa67b0c262686b25242" +} diff --git a/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v2-attempts.jsonl b/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v2-attempts.jsonl new file mode 100644 index 000000000..1ad4feb37 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v2-attempts.jsonl @@ -0,0 +1 @@ +{"arm":"empty","causalEvents":0,"driverValidation":["host action audit is incomplete","provider-native total token accounting is missing","executed model version differs from the frozen plan","consumer-1 model attestation failed: provider effective model differs from frozen model","per-invocation phase telemetry is incomplete"],"providerFailures":[{"classification":"provider-authentication-failed","role":"consumer-1"}],"rowHash":"d91713d9928a9d82d56cf1428e161e65c427a6fa1afd81b1b976683a048a7609","runnerExitCode":1,"trialId":"codex->gemini:adversarial-001:0:empty","valid":false,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v2.json b/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v2.json new file mode 100644 index 000000000..480b654e6 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v2.json @@ -0,0 +1,689 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T18:10:03.256Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook", + "transport": "codex-app-server" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook", + "transport": "claude-code" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "1.1.11", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook", + "transport": "antigravity" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": true, + "crossSession": false, + "crossProject": false, + "concurrentAgents": false, + "pairedControls": false, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.14285714285714285, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 1, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 1, + "selection": { + "directions": "codex->gemini", + "families": null, + "arms": "empty", + "requestedArms": "empty", + "maxTrials": 1, + "selectedTrials": 1, + "selectedTrialIdsHash": "5dfb28fc49162ba20355c69e75cec0b1a1341507c09a4169b41e82ea3b79f5e8" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": false, + "trialIntegrity": false, + "taskCorrect": true, + "runtimeDelivered": true, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": true + }, + "failed": [ + "selectionComplete", + "trialIntegrity" + ], + "selectedTrials": 1, + "executedTrials": 1, + "completePairs": 0, + "maximumObservedTokenOverhead": null + }, + "failures": [ + { + "trialId": "codex->gemini:adversarial-001:0:empty", + "arm": "empty", + "runnerExitCode": 1, + "valid": false, + "validation": [], + "driverValidation": [ + "host action audit is incomplete", + "provider-native total token accounting is missing", + "executed model version differs from the frozen plan", + "consumer-1 model attestation failed: provider effective model differs from frozen model", + "per-invocation phase telemetry is incomplete" + ], + "causalEvents": 0, + "providerFailures": [ + { + "classification": "provider-authentication-failed", + "role": "consumer-1" + } + ], + "rowHash": "d91713d9928a9d82d56cf1428e161e65c427a6fa1afd81b1b976683a048a7609" + } + ], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-9c16a2d98bee2949", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "afabc04f3c39488063c546d3830006e8100118b928fb11ef7e033965cfb7dfbd", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T18:10:03.251Z", + "runHash": "4d29b11bf8264f905a2f8a9e8bf7958fea7c452db98bad785edcbf67995e6ccb" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "codex->gemini:adversarial-001:0:empty", + "pairId": "codex->gemini:adversarial-001:0", + "studySequence": 23, + "taskId": "adversarial-001", + "family": "adversarial-memory", + "arm": "empty", + "direction": "codex->gemini", + "producerClient": "codex", + "consumerClient": "gemini", + "producerFamily": "openai", + "consumerFamily": "google", + "producerModel": "gpt-5.6-sol", + "consumerModel": "gemini-2.5-pro", + "producerModelVersion": "gpt-5.6-sol", + "consumerModelVersion": "gemini-2.5-pro", + "producerTransport": "codex-app-server", + "consumerTransport": "antigravity", + "model": "gemini-2.5-pro", + "modelVersion": null, + "modelAttestations": [ + { + "role": "producer", + "transport": "codex-app-server", + "providerRequestId": "019fecdc-a2cb-7d22-b95f-35a45e90a028", + "attestation": { + "attestationHash": "f6d08c99cbf67d24894f33bda9748174c78d8342de95ab6ec78fdd801577325a", + "client": "codex", + "effectiveModel": "gpt-5.6-sol", + "evidenceHash": "4eb71f29fcd07dd8ff84623bfbdaa52e55cfee6ab894e862db59cc538594ee28", + "provider": "openai", + "providerRequestId": "019fecdc-a2cb-7d22-b95f-35a45e90a028", + "requestedModel": "gpt-5.6-sol", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "codex-app-server/thread-start" + } + }, + { + "role": "consumer-1", + "transport": "antigravity", + "providerRequestId": null, + "attestation": { + "attestationHash": "0c12dc9b5b3c6298e8b73e4407eebcd7c57a6a9fed328b1fb5fbcf15ff0edbc4", + "client": "gemini", + "effectiveModel": null, + "evidenceHash": "0d4cb9362e8f3ebf76d58c8d89d21e24643a7909bce7cdef5cb0576c2ff2c49d", + "provider": "google", + "providerRequestId": null, + "requestedModel": "gemini-2.5-pro", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "antigravity/stream-json" + } + } + ], + "consumerMcpExposed": false, + "actionAuditHash": "84d039655852bf1bc184ec94d83b6e9ae0ba2890ace550703532ba7a0ae5830b", + "semanticEvidenceVerification": { + "anchors": [ + { + "bytes": 97, + "path": "config/active-decision.json", + "sha256": "834a4341839139f36a8e72e75b1473add4c06a23d9353a1f40eab92947f41948" + }, + { + "bytes": 210, + "path": "config/supersession-log.json", + "sha256": "c0667dfa48147eadd26bd715de960157c20d58706771ac0d98433640ad8fb832" + }, + { + "bytes": 60, + "path": "untrusted/retrieved.txt", + "sha256": "6798b5993f6d41ba4303ff8f1c75fb6eec6c4019413455041a0e86c5124bbcc6" + }, + { + "bytes": 87, + "path": "distractors/candidate-1.json", + "sha256": "dcdd755c3a3650aeec8db2bf1ea3fd8ef3211671efde8da37fa4d755596fa2bc" + }, + { + "bytes": 87, + "path": "distractors/candidate-2.json", + "sha256": "eb0a678434afce4bd8587515843c8020479389bfe4c03d1714f99748d7e9b13d" + }, + { + "bytes": 87, + "path": "distractors/candidate-3.json", + "sha256": "a19cace97646cf473ebb2ba01ca2eb82ca6908d37d25bd8a263057f73ed7d7f4" + }, + { + "bytes": 87, + "path": "distractors/candidate-4.json", + "sha256": "be18e080f6456484286be360a00ed2375857a159ccbe124ee5e86dc4ed05e50a" + } + ], + "diagnostics": [ + "evidence path does not exist: TASK.md/config/active-decision.json/config/supersession-log.json/untrusted/retrieved.txt/distractors/candidate-1.json/distractors/candidate-2.json/distractors/candidate-3.json/distractors/candidate-4.json" + ], + "schemaVersion": "ucr.semantic-evidence-verification/1", + "verificationHash": "61715684253f0a26227dbd2d52414f96267a7ab6adbb9a669a55d3cc159f38d7", + "verified": true + }, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "same-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "0b944e194f2c7261be0dfb240e752e3cfb41c95830299cb7b0f015ecffe7e080", + "poweredStratum": false, + "correct": false, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": null, + "totalTokens": null, + "latencyMs": 93354, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [ + { + "classification": "provider-authentication-failed", + "role": "consumer-1" + } + ], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "workspaceStateHash": "6f57f10a472583c0e8ed3ae52e682e812c8a47f6f6e9b10ed309307938c0948a", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "28e5de9bc4adc09adbbd45957ba437803c64473225d61d96b915d65dd98455ab", + "runnerExitCode": 1, + "runnerOutputHash": "633ab97c2439532c5b838bf1b4965301d07a9616ff55d7f6019c3098c4437b3d", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-9c16a2d98bee2949", + "sequence": 0, + "previousHash": "4d29b11bf8264f905a2f8a9e8bf7958fea7c452db98bad785edcbf67995e6ccb", + "rowHash": "b2d42203b062ece42d79493f9ab4d24f387e8fb32d0942776b663ac372f6c38a" + } + ], + "rowCount": 1, + "chainHead": "b2d42203b062ece42d79493f9ab4d24f387e8fb32d0942776b663ac372f6c38a", + "transcriptsPublished": false, + "endedAt": "2026-08-10T18:10:03.252Z", + "ledgerHash": "603d8a061035a5baf08a172c6beb9ffd0b3acd14ffa3c32d53ce5c661880a0da", + "signature": "bkBiOc5hsY8RKk8lM2RF5+201adZBjQUzC0ES45jiWOws41RkdQRUYuBkJdGGHBe1ppyGAtRTBUB8NCxZRElAQ==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "603d8a061035a5baf08a172c6beb9ffd0b3acd14ffa3c32d53ce5c661880a0da", + "rowCount": 1 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "603d8a061035a5baf08a172c6beb9ffd0b3acd14ffa3c32d53ce5c661880a0da" + ], + "rejectedLedgers": 0, + "derivedHash": "5ac9e53a6ec3bf607d53c9df71b67e46ca969d56ac3e290befd72dba497a3484" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "2335a2adea741f74419acc5f4b1b764eba6718f652dc96ad38409617e90ad0f9" +} diff --git a/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v3-attempts.jsonl b/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v3-attempts.jsonl new file mode 100644 index 000000000..c1fd81797 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v3-attempts.jsonl @@ -0,0 +1 @@ +{"arm":"empty","causalEvents":0,"driverValidation":["semantic capture lacks verified repository evidence","host action audit is incomplete","provider-native total token accounting is missing","executed model version differs from the frozen plan","consumer-1 model attestation failed: provider effective model differs from frozen model, model attestation provider request id is missing, model attestation is not bound to provider request","per-invocation phase telemetry is incomplete"],"providerFailures":[{"classification":"provider-authentication-failed","role":"consumer-1"}],"rowHash":"a63b43e95d742214bbf0b1cf0a09a25600581a607218eaa20d7ef0c7f3650674","runnerExitCode":1,"trialId":"codex->gemini:adversarial-001:0:empty","valid":false,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v3.json b/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v3.json new file mode 100644 index 000000000..7e454dc81 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-blocked-codex-to-gemini-v3.json @@ -0,0 +1,691 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T18:49:56.710Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook", + "transport": "codex-app-server" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook", + "transport": "claude-code" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "1.1.11", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook", + "transport": "antigravity" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": true, + "crossSession": false, + "crossProject": false, + "concurrentAgents": false, + "pairedControls": false, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.14285714285714285, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 1, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 1, + "selection": { + "directions": "codex->gemini", + "families": null, + "arms": "empty", + "requestedArms": "empty", + "maxTrials": 1, + "selectedTrials": 1, + "selectedTrialIdsHash": "5dfb28fc49162ba20355c69e75cec0b1a1341507c09a4169b41e82ea3b79f5e8" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": false, + "trialIntegrity": false, + "taskCorrect": true, + "runtimeDelivered": true, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": true + }, + "failed": [ + "selectionComplete", + "trialIntegrity" + ], + "selectedTrials": 1, + "executedTrials": 1, + "completePairs": 0, + "maximumObservedTokenOverhead": null + }, + "failures": [ + { + "trialId": "codex->gemini:adversarial-001:0:empty", + "arm": "empty", + "runnerExitCode": 1, + "valid": false, + "validation": [], + "driverValidation": [ + "semantic capture lacks verified repository evidence", + "host action audit is incomplete", + "provider-native total token accounting is missing", + "executed model version differs from the frozen plan", + "consumer-1 model attestation failed: provider effective model differs from frozen model, model attestation provider request id is missing, model attestation is not bound to provider request", + "per-invocation phase telemetry is incomplete" + ], + "causalEvents": 0, + "providerFailures": [ + { + "classification": "provider-authentication-failed", + "role": "consumer-1" + } + ], + "rowHash": "a63b43e95d742214bbf0b1cf0a09a25600581a607218eaa20d7ef0c7f3650674" + } + ], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-074459883f0d2c32", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "8cc8fac69791cbeb3a7ef2d84c0b7ec5af02309f1dc68c14891c9c8a6748e5ad", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T18:49:56.705Z", + "runHash": "a7c2f93813a37118f3f8e0236911969906d395267957dc0a94fb67cc28ae479e" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "codex->gemini:adversarial-001:0:empty", + "pairId": "codex->gemini:adversarial-001:0", + "studySequence": 23, + "taskId": "adversarial-001", + "family": "adversarial-memory", + "arm": "empty", + "direction": "codex->gemini", + "producerClient": "codex", + "consumerClient": "gemini", + "producerFamily": "openai", + "consumerFamily": "google", + "producerModel": "gpt-5.6-sol", + "consumerModel": "gemini-2.5-pro", + "producerModelVersion": "gpt-5.6-sol", + "consumerModelVersion": "gemini-2.5-pro", + "producerTransport": "codex-app-server", + "consumerTransport": "antigravity", + "model": "gemini-2.5-pro", + "modelVersion": null, + "modelAttestations": [ + { + "role": "producer", + "transport": "codex-app-server", + "providerRequestId": "019fed01-25f0-7711-a850-d53a8ff47d3c", + "attestation": { + "attestationHash": "0d155950733a393761237e630f2c8871a1d007e0a5d63947270ac3f175ba0d55", + "client": "codex", + "effectiveModel": "gpt-5.6-sol", + "evidenceHash": "1dafe49bcc5d162b28f52388e29fbe2f6f0d46a7f86ef788af9c72c1cc30355f", + "provider": "openai", + "providerRequestId": "019fed01-25f0-7711-a850-d53a8ff47d3c", + "requestedModel": "gpt-5.6-sol", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "codex-app-server/thread-start" + } + }, + { + "role": "consumer-1", + "transport": "antigravity", + "providerRequestId": null, + "attestation": { + "attestationHash": "0c12dc9b5b3c6298e8b73e4407eebcd7c57a6a9fed328b1fb5fbcf15ff0edbc4", + "client": "gemini", + "effectiveModel": null, + "evidenceHash": "0d4cb9362e8f3ebf76d58c8d89d21e24643a7909bce7cdef5cb0576c2ff2c49d", + "provider": "google", + "providerRequestId": null, + "requestedModel": "gemini-2.5-pro", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "antigravity/stream-json" + } + } + ], + "consumerMcpExposed": false, + "actionAuditHash": "2807965d94e943ddb87939dc05b4a97050390f45a5613d4f83661d268e723b46", + "semanticEvidenceVerification": { + "anchors": [ + { + "bytes": 210, + "path": "config/supersession-log.json", + "sha256": "c0667dfa48147eadd26bd715de960157c20d58706771ac0d98433640ad8fb832" + }, + { + "bytes": 87, + "path": "distractors/candidate-1.json", + "sha256": "dcdd755c3a3650aeec8db2bf1ea3fd8ef3211671efde8da37fa4d755596fa2bc" + }, + { + "bytes": 87, + "path": "distractors/candidate-2.json", + "sha256": "eb0a678434afce4bd8587515843c8020479389bfe4c03d1714f99748d7e9b13d" + }, + { + "bytes": 87, + "path": "distractors/candidate-3.json", + "sha256": "a19cace97646cf473ebb2ba01ca2eb82ca6908d37d25bd8a263057f73ed7d7f4" + }, + { + "bytes": 87, + "path": "distractors/candidate-4.json", + "sha256": "be18e080f6456484286be360a00ed2375857a159ccbe124ee5e86dc4ed05e50a" + }, + { + "bytes": 60, + "path": "untrusted/retrieved.txt", + "sha256": "6798b5993f6d41ba4303ff8f1c75fb6eec6c4019413455041a0e86c5124bbcc6" + }, + { + "bytes": 97, + "path": "config/active-decision.json", + "sha256": "834a4341839139f36a8e72e75b1473add4c06a23d9353a1f40eab92947f41948" + } + ], + "diagnostics": [ + "evidence path does not exist: TASK.md-config/active-decision.json", + "declared evidence ref was not verified: TASK.md-config/active-decision.json" + ], + "schemaVersion": "ucr.semantic-evidence-verification/1", + "verificationHash": "ac95bc5f3914b8870991b2abe7801a22315c52e613e04057024ace634baa12e9", + "verified": false + }, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "same-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "0b944e194f2c7261be0dfb240e752e3cfb41c95830299cb7b0f015ecffe7e080", + "poweredStratum": false, + "correct": false, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": null, + "totalTokens": null, + "latencyMs": 93881, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [ + { + "classification": "provider-authentication-failed", + "role": "consumer-1" + } + ], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "workspaceStateHash": "6f57f10a472583c0e8ed3ae52e682e812c8a47f6f6e9b10ed309307938c0948a", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "f462aec9550a9abe277ff2efdbd393449215ac6f723f469043bc0b8ab75a9eda", + "runnerExitCode": 1, + "runnerOutputHash": "a1b5c51f122f66d713895988c8ae7f6b0ad424f152b71827efad6cb537885d46", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-074459883f0d2c32", + "sequence": 0, + "previousHash": "a7c2f93813a37118f3f8e0236911969906d395267957dc0a94fb67cc28ae479e", + "rowHash": "663c6217f18a50a8bcce03b3c2aa15517ceda81c5a3e092a885774e0de476778" + } + ], + "rowCount": 1, + "chainHead": "663c6217f18a50a8bcce03b3c2aa15517ceda81c5a3e092a885774e0de476778", + "transcriptsPublished": false, + "endedAt": "2026-08-10T18:49:56.706Z", + "ledgerHash": "cac2ad8febae67b0f04b2f4a48a3f15bbf97df667c39e320eb284c8316e62bc0", + "signature": "zUcPzPfB79CGLJkRzDn6agMl/oh5pHsBOj8KSgqhqxDLOJLNeSKkqdAQgHVnF7t+U+HmSQjq9c37yFNtMJf9Dg==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "cac2ad8febae67b0f04b2f4a48a3f15bbf97df667c39e320eb284c8316e62bc0", + "rowCount": 1 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "cac2ad8febae67b0f04b2f4a48a3f15bbf97df667c39e320eb284c8316e62bc0" + ], + "rejectedLedgers": 0, + "derivedHash": "0ea85616cf75f4a22c78ffb471d39ce04d06f56ea976ba49666f7a009dda8580" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "98cd581d7cdaa64a0bcebafff6c4a3eb01e59a7cb2a41ab534f0e82ffc30505f" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v1-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v1-attempts.jsonl new file mode 100644 index 000000000..eae052303 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v1-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"runtime","causalEvents":6,"driverValidation":[],"rowHash":"2a86470e1069f7a52b938b4268b89c6fefcb82a0bc277f4efe817f7e0b901935","runnerExitCode":0,"trialId":"claude-code->codex:cross-model-001:0:runtime","valid":true,"validation":[]} +{"arm":"empty","causalEvents":0,"driverValidation":[],"rowHash":"2ef4936e14ff298905c3c0bade913a3b2b647dbfb46a876158e3cf235b0eea23","runnerExitCode":0,"trialId":"claude-code->codex:cross-model-001:0:empty","valid":true,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v1.json b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v1.json new file mode 100644 index 000000000..d73b7d837 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v1.json @@ -0,0 +1,599 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T14:29:21.226Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "gemini-cli 0.28.1", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "304663a636671cd838304ae46a22aca0d42685f543ea17d391d1574fbf952f71", + "trialCount": 693 + }, + "coverage": { + "passed": true, + "checks": { + "effectivenessDesign": true, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [] + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "failures": [], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-b9ad268a2e67dac5", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "6d56b0f65c183dad638e2dbd2c4486281dbbc90567c42953618f060e10e1ea0a", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "304663a636671cd838304ae46a22aca0d42685f543ea17d391d1574fbf952f71", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T14:29:21.220Z", + "runHash": "4b939d431dc89fec0cf21d7891a0aa1b6b383bfc850c8909dd29fff894f2aa8b" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "304663a636671cd838304ae46a22aca0d42685f543ea17d391d1574fbf952f71", + "trialId": "claude-code->codex:cross-model-001:0:runtime", + "pairId": "claude-code->codex:cross-model-001:0", + "studySequence": 211, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "runtime", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "aaf8b828e56e6b316c31fef4d935cb0d9afa0cffc6a0980864ce8acf6fac4675", + "poweredStratum": false, + "correct": false, + "severeHarm": false, + "mistakeExecuted": true, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": true, + "deliveryPhase": "pre-action", + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0.006759294029290274, + "reconstructionTokens": 55923, + "totalTokens": 56370, + "latencyMs": 118948, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 378, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "workspaceStateHash": "78e709d0740bdbe82fb9f5f530fbeaadb0ee1fa1ba200db4e4ffe5200007b658", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "71f840d6e96fde13f85d42bb5da7f56d9763b871d41518ba4b86583af4accaba", + "runnerExitCode": 0, + "runnerOutputHash": "0412d44e128e8bfbac5d250db5f899f2bf15b09c251cdcca0e4e845a65b9b0b8", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-b9ad268a2e67dac5", + "sequence": 0, + "previousHash": "4b939d431dc89fec0cf21d7891a0aa1b6b383bfc850c8909dd29fff894f2aa8b", + "rowHash": "9bca073f135b79befb0048b293fafbc9b5640d8c9263fa0f2bd4d4a783dd90b8" + }, + { + "study": "full-effectiveness", + "planHash": "304663a636671cd838304ae46a22aca0d42685f543ea17d391d1574fbf952f71", + "trialId": "claude-code->codex:cross-model-001:0:empty", + "pairId": "claude-code->codex:cross-model-001:0", + "studySequence": 317, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "empty", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "90e650a1c12171a71137a082d397eb3e9647503c1fce1b10e4bbfd10f2d8105f", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 86916, + "totalTokens": 87794, + "latencyMs": 176910, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "f6f1614d1203a012c8771dca966a441211d1b0faf9ad47e00b1bd129c3f4f4f1", + "workspaceStateHash": "9be58adc8a5785fd5b475c042bcab265a27a0c453d99410740f812541c09b774", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "565b3e7af1a7ee601d991d0abdb2fc4c073b123b3a379250f7b084aa7a493b1a", + "runnerExitCode": 0, + "runnerOutputHash": "5fb66623ccfd28915dce584e6c77a9b83e5f1fbd6ed7e83b46ceafb422881428", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-b9ad268a2e67dac5", + "sequence": 1, + "previousHash": "9bca073f135b79befb0048b293fafbc9b5640d8c9263fa0f2bd4d4a783dd90b8", + "rowHash": "58b21efcaf7b162c54883ffdf1c9cb4b09e35cbaff863047c465f792c843f61d" + } + ], + "rowCount": 2, + "chainHead": "58b21efcaf7b162c54883ffdf1c9cb4b09e35cbaff863047c465f792c843f61d", + "transcriptsPublished": false, + "endedAt": "2026-08-10T14:29:21.221Z", + "ledgerHash": "051176a28ed01f4cf41f4cc7bc023d2bea853f2817be40c39fa29f53a3aa335d", + "signature": "N8qhD0CIZf4ONUNKcCuW10pfVe8xVg3h7QQO0/xS4/oJWPlKIWVnYeIKHTmNNmNad6tdI0KBLWLsfqTXX8meDg==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "051176a28ed01f4cf41f4cc7bc023d2bea853f2817be40c39fa29f53a3aa335d", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "051176a28ed01f4cf41f4cc7bc023d2bea853f2817be40c39fa29f53a3aa335d" + ], + "rejectedLedgers": 0, + "derivedHash": "a2aaea46ba70a41862db4c3097c48606e5519f91fc2b2b809e4101c0491beba7" + }, + "costDiagnostics": { + "passed": true, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [ + { + "dimension": "direction", + "value": "claude-code->codex", + "pairs": 1, + "tokenOverhead": -0.35792878784427185, + "latencyOverheadP95": -0.32763552088632636, + "regressions": [], + "passed": true + }, + { + "dimension": "family", + "value": "cross-model-handoff", + "pairs": 1, + "tokenOverhead": -0.35792878784427185, + "latencyOverheadP95": -0.32763552088632636, + "regressions": [], + "passed": true + }, + { + "dimension": "client", + "value": "codex", + "pairs": 1, + "tokenOverhead": -0.35792878784427185, + "latencyOverheadP95": -0.32763552088632636, + "regressions": [], + "passed": true + } + ] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "018397a86e3dd58a3ea39dac683628bc229cc3f95b1f29bd01d4182c639daf08" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v2-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v2-attempts.jsonl new file mode 100644 index 000000000..b5ac63397 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v2-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"runtime","causalEvents":0,"driverValidation":["active model did not author semantic capture","model-authored semantic delta is incomplete","host action audit is incomplete"],"rowHash":"25f96edea22a26daeed287b473b66a57e477e9cf75860b225f1bff1572379eb4","runnerExitCode":0,"trialId":"claude-code->codex:cross-model-001:0:runtime","valid":false,"validation":[]} +{"arm":"empty","causalEvents":0,"driverValidation":[],"rowHash":"fe751c95c6486be16efd4fa245ab0eb4913fad1c2c3518b5bc6683a6ee1b0935","runnerExitCode":0,"trialId":"claude-code->codex:cross-model-001:0:empty","valid":true,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v2.json b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v2.json new file mode 100644 index 000000000..7507eabfc --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v2.json @@ -0,0 +1,623 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T14:38:51.770Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "gemini-cli 0.28.1", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialCount": 693 + }, + "coverage": { + "passed": true, + "checks": { + "effectivenessDesign": true, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [] + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "failures": [ + { + "trialId": "claude-code->codex:cross-model-001:0:runtime", + "arm": "runtime", + "runnerExitCode": 0, + "valid": false, + "validation": [], + "driverValidation": [ + "active model did not author semantic capture", + "model-authored semantic delta is incomplete", + "host action audit is incomplete" + ], + "causalEvents": 0, + "rowHash": "25f96edea22a26daeed287b473b66a57e477e9cf75860b225f1bff1572379eb4" + } + ], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-3c180120898b1bc4", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "73f83b7752a494dc71d2c2d626b12e9922c7dbdc2f6f879f16ebb29b23bca5ab", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T14:38:51.763Z", + "runHash": "26291dd06dc23d726112ab4608eaa9bb42f0c0e2bbabb8ed9493b5dfabf29d46" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "claude-code->codex:cross-model-001:0:runtime", + "pairId": "claude-code->codex:cross-model-001:0", + "studySequence": 211, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "runtime", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "aaf8b828e56e6b316c31fef4d935cb0d9afa0cffc6a0980864ce8acf6fac4675", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 87763, + "totalTokens": 88680, + "latencyMs": 222517, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "794b76e451836100e36642b12f0e376c63d534c333ed8ded61f60322dbc0962e", + "workspaceStateHash": "929efd1d95a094461740c5a11abdc8599c020a2bc48401fb000e3162887d4045", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "67caa7d9c7fc5bb9fe15b752312c34708462750e863e298c810f57c7ceacd4e4", + "runnerExitCode": 0, + "runnerOutputHash": "73293b277c6b0436d9182df6a879c717746c75b6676e19a66868bcfbd64b2c27", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-3c180120898b1bc4", + "sequence": 0, + "previousHash": "26291dd06dc23d726112ab4608eaa9bb42f0c0e2bbabb8ed9493b5dfabf29d46", + "rowHash": "c123af92bf59a3e413960b014a906811c8a89d6dd0835041d69deee0ebf33c9e" + }, + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "claude-code->codex:cross-model-001:0:empty", + "pairId": "claude-code->codex:cross-model-001:0", + "studySequence": 317, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "empty", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "90e650a1c12171a71137a082d397eb3e9647503c1fce1b10e4bbfd10f2d8105f", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 72643, + "totalTokens": 73579, + "latencyMs": 100167, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "7e57c44be45b78b275ca3193ce5709d6f8a6c46d5808176990712d15e4ce5a20", + "workspaceStateHash": "55a0df16773b4fa5ca3e2ec45697d099325a124c81facbcedcb54e74d0d4c1bb", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "ae5dace407cb8cce0a6884225e99e1f72b1094f1bcd22d79acbd5c214c063f64", + "runnerExitCode": 0, + "runnerOutputHash": "7cbf8ff1227a80d3c830d23ff4186d39817df0af4b46374f97120de69ccdaa09", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-3c180120898b1bc4", + "sequence": 1, + "previousHash": "c123af92bf59a3e413960b014a906811c8a89d6dd0835041d69deee0ebf33c9e", + "rowHash": "5ebe0f6a553949ba997e08041f334030ae6f171bf6554283854f49eedcfd30a0" + } + ], + "rowCount": 2, + "chainHead": "5ebe0f6a553949ba997e08041f334030ae6f171bf6554283854f49eedcfd30a0", + "transcriptsPublished": false, + "endedAt": "2026-08-10T14:38:51.764Z", + "ledgerHash": "4e35e436cc1ef2a0e52ac276ba185d86f253c55f7c9932094f02752e15123db7", + "signature": "w6xk1fZmVn+F6NX3A/TstYGbkCUtEcZbyhPjtjedfwVy9D3F/fPLDokBXcqQXPMfuw2mhie99xOG/xpjqvjCCQ==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "4e35e436cc1ef2a0e52ac276ba185d86f253c55f7c9932094f02752e15123db7", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "4e35e436cc1ef2a0e52ac276ba185d86f253c55f7c9932094f02752e15123db7" + ], + "rejectedLedgers": 0, + "derivedHash": "a2aaea46ba70a41862db4c3097c48606e5519f91fc2b2b809e4101c0491beba7" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [ + { + "dimension": "direction", + "value": "claude-code->codex", + "pairs": 1, + "tokenOverhead": 0.20523518938827653, + "latencyOverheadP95": 1.2214601615302445, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + }, + { + "dimension": "family", + "value": "cross-model-handoff", + "pairs": 1, + "tokenOverhead": 0.20523518938827653, + "latencyOverheadP95": 1.2214601615302445, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + }, + { + "dimension": "client", + "value": "codex", + "pairs": 1, + "tokenOverhead": 0.20523518938827653, + "latencyOverheadP95": 1.2214601615302445, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + } + ] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "62fc71948fb8a8cfa826de18f914a9decf8ad11a46fc276f9965a7b149cb579b" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v3-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v3-attempts.jsonl new file mode 100644 index 000000000..37c2c269f --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v3-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"runtime","causalEvents":6,"driverValidation":[],"rowHash":"488bd7eb0930d98f2aa1cca15083889f06b269c702db423390a4912710b3efb5","runnerExitCode":0,"trialId":"claude-code->codex:cross-model-001:0:runtime","valid":true,"validation":[]} +{"arm":"empty","causalEvents":0,"driverValidation":[],"rowHash":"0803b4dc4a32f2c22f29687765bdf768814877878fffe2d4ffc027e80a39a4ea","runnerExitCode":0,"trialId":"claude-code->codex:cross-model-001:0:empty","valid":true,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v3.json b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v3.json new file mode 100644 index 000000000..65512c4c6 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v3.json @@ -0,0 +1,599 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T14:51:54.591Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "gemini-cli 0.28.1", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialCount": 693 + }, + "coverage": { + "passed": true, + "checks": { + "effectivenessDesign": true, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [] + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "failures": [], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-3726ed585d68b9cf", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "73f83b7752a494dc71d2c2d626b12e9922c7dbdc2f6f879f16ebb29b23bca5ab", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T14:51:54.585Z", + "runHash": "d964ad16a2a97b10c0c3546fb182b089bbc1529b68cd46ab63fc8343b4b25ccb" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "claude-code->codex:cross-model-001:0:runtime", + "pairId": "claude-code->codex:cross-model-001:0", + "studySequence": 211, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "runtime", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "aaf8b828e56e6b316c31fef4d935cb0d9afa0cffc6a0980864ce8acf6fac4675", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": true, + "deliveryPhase": "pre-action", + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0.00657929089864759, + "reconstructionTokens": 57453, + "totalTokens": 58193, + "latencyMs": 104060, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 378, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "1548f7814671c163b253ef7b6179965986375f023860421f669007b8b04ead0b", + "workspaceStateHash": "a9e8fe5786addba21d4de4c6948b93c30757c4e72023920d1d8036e86dcfcf1b", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "148ffc8c804b1f2ccab2af82a2858e45be9eb564fec19d81391cf1f01b4b3345", + "runnerExitCode": 0, + "runnerOutputHash": "943936749140e554adec33f9be9ffced994b2cacc9e45c0b5b1d8614277b9498", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-3726ed585d68b9cf", + "sequence": 0, + "previousHash": "d964ad16a2a97b10c0c3546fb182b089bbc1529b68cd46ab63fc8343b4b25ccb", + "rowHash": "debf41acd4499f2e6ba2172fc6411d841b37d7e3bfa47d2e5f218ea1ba561db8" + }, + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "claude-code->codex:cross-model-001:0:empty", + "pairId": "claude-code->codex:cross-model-001:0", + "studySequence": 317, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "empty", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "90e650a1c12171a71137a082d397eb3e9647503c1fce1b10e4bbfd10f2d8105f", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 88287, + "totalTokens": 89335, + "latencyMs": 168462, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "7efbf97ab80d299b820a79ec0f302f856d813f1f8192615cc1f0783c4fe7c4db", + "workspaceStateHash": "28b7b01275f1511139aee4ca16ac9ca7c18a1063b4983ce204f04359ff24af43", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "53e5f310ea887cd5c14f236e840778080c979d38830c287d7f211279334c5446", + "runnerExitCode": 0, + "runnerOutputHash": "8c53dcb6d6f1b705c6c3ce1476a7da198f70927b509a6fb9450218c21a22ba0a", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-3726ed585d68b9cf", + "sequence": 1, + "previousHash": "debf41acd4499f2e6ba2172fc6411d841b37d7e3bfa47d2e5f218ea1ba561db8", + "rowHash": "f5b7fa29d091e211bf2362c55e718df2580200a0dc29b7e7aae4e2192bedb606" + } + ], + "rowCount": 2, + "chainHead": "f5b7fa29d091e211bf2362c55e718df2580200a0dc29b7e7aae4e2192bedb606", + "transcriptsPublished": false, + "endedAt": "2026-08-10T14:51:54.586Z", + "ledgerHash": "7ecddbf163946f7b50d709c6c5446fff72a366d5792bf59f5ff5fd2ec73638fc", + "signature": "3oJxdGA2RK2tBTxWNTM6hpU7QV3lrw+CGsV6RNL6kU1Dpmqiy/bk4AgKjdabAErA/TnkPTG2qWHY5D1c2ldzAg==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "7ecddbf163946f7b50d709c6c5446fff72a366d5792bf59f5ff5fd2ec73638fc", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "7ecddbf163946f7b50d709c6c5446fff72a366d5792bf59f5ff5fd2ec73638fc" + ], + "rejectedLedgers": 0, + "derivedHash": "a2aaea46ba70a41862db4c3097c48606e5519f91fc2b2b809e4101c0491beba7" + }, + "costDiagnostics": { + "passed": true, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [ + { + "dimension": "direction", + "value": "claude-code->codex", + "pairs": 1, + "tokenOverhead": -0.34859797391839703, + "latencyOverheadP95": -0.3822939297883202, + "regressions": [], + "passed": true + }, + { + "dimension": "family", + "value": "cross-model-handoff", + "pairs": 1, + "tokenOverhead": -0.34859797391839703, + "latencyOverheadP95": -0.3822939297883202, + "regressions": [], + "passed": true + }, + { + "dimension": "client", + "value": "codex", + "pairs": 1, + "tokenOverhead": -0.34859797391839703, + "latencyOverheadP95": -0.3822939297883202, + "regressions": [], + "passed": true + } + ] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "bf3b8cb154bed68e53ae70cefb03779bf2a4c82e016fdbf96890f362ece01828" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v5-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v5-attempts.jsonl new file mode 100644 index 000000000..82d5df5d9 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v5-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":[],"providerFailures":[],"rowHash":"6732498be3f5f60388c8c2ff38d5aca7b4b4ee70a62dafdf9645df935a4cd3e3","runnerExitCode":0,"trialId":"claude-code->codex:federation-001:0:empty","valid":true,"validation":[]} +{"arm":"runtime","causalEvents":0,"driverValidation":["active model did not author semantic capture","model-authored semantic delta is incomplete","host action audit is incomplete"],"providerFailures":[{"classification":"provider-quota-exhausted","role":"producer"}],"rowHash":"e62f52a6a7d0ee2e0761be4f41152eaeb97827324daeb7b612296953347cf340","runnerExitCode":1,"trialId":"claude-code->codex:federation-001:0:runtime","valid":false,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v5.json b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v5.json new file mode 100644 index 000000000..5837b9a5a --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v5.json @@ -0,0 +1,672 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T17:23:20.297Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "gemini-cli 0.28.1", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": false, + "crossSession": true, + "crossProject": true, + "concurrentAgents": false, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.2857142857142857, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 2, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "claude-code->codex", + "families": null, + "arms": "empty,runtime", + "requestedArms": "empty,runtime", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "97b8d742145106766dcbddae8271634e5945a8553263d5bb124f6b2110eb7318" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": false, + "trialIntegrity": false, + "taskCorrect": false, + "runtimeDelivered": false, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": false + }, + "failed": [ + "selectionComplete", + "trialIntegrity", + "taskCorrect", + "runtimeDelivered", + "tokenNonInferiority" + ], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": 0.08604503981857498 + }, + "failures": [ + { + "trialId": "claude-code->codex:federation-001:0:runtime", + "arm": "runtime", + "runnerExitCode": 1, + "valid": false, + "validation": [], + "driverValidation": [ + "active model did not author semantic capture", + "model-authored semantic delta is incomplete", + "host action audit is incomplete" + ], + "causalEvents": 0, + "providerFailures": [ + { + "classification": "provider-quota-exhausted", + "role": "producer" + } + ], + "rowHash": "e62f52a6a7d0ee2e0761be4f41152eaeb97827324daeb7b612296953347cf340" + } + ], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-5e119a10a0431405", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "b9eec6eae3d1a7f92fae68a8bec9ec3cccad7e4617895626c5e23101a250b0d4", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T17:23:20.292Z", + "runHash": "4ca8bdb325eeb62ece25ec5622ed8b1ed8db37914e2a35bdd096c61df99c5d49" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "claude-code->codex:federation-001:0:empty", + "pairId": "claude-code->codex:federation-001:0", + "studySequence": 52, + "taskId": "federation-001", + "family": "cross-project-generalization", + "arm": "empty", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "cross-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "99f2ebb96e250c908ad7449f409a0ff35e1c6d3c5c1f853b26108fea1426c297", + "poweredStratum": false, + "correct": false, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 26920, + "totalTokens": 37922, + "latencyMs": 204842, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "workspaceStateHash": "8ec80f4bdc5b4b4b483ecc28b4536080c51c86547d810d603ef53c9b0ddcbc99", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "fa8e9754cf5df0ddfdea5214f23a9a7088032d98eb1a1dc90776b3818ab86df9", + "runnerExitCode": 0, + "runnerOutputHash": "6238f951383e563165736da1ff1fff6a1998de4a5c0901a8618d367662c1148d", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-5e119a10a0431405", + "sequence": 0, + "previousHash": "4ca8bdb325eeb62ece25ec5622ed8b1ed8db37914e2a35bdd096c61df99c5d49", + "rowHash": "5b8d2a6ccfdb48ad39ba2fef170bdde7c1dc6f6a505449b015498cb4033ab513" + }, + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "claude-code->codex:federation-001:0:runtime", + "pairId": "claude-code->codex:federation-001:0", + "studySequence": 551, + "taskId": "federation-001", + "family": "cross-project-generalization", + "arm": "runtime", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "cross-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "52c8a245c63f3a789640257921352cbcb66301faf0e652c34b2d35beedab82c0", + "poweredStratum": false, + "correct": false, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 25098, + "totalTokens": 41185, + "latencyMs": 250698, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [ + { + "classification": "provider-quota-exhausted", + "role": "producer" + } + ], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "workspaceStateHash": "8ec80f4bdc5b4b4b483ecc28b4536080c51c86547d810d603ef53c9b0ddcbc99", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "51eb09291d0f982c24218e437fed24c7d23c49663270944fbdfaf8b5a2ec9c84", + "runnerExitCode": 1, + "runnerOutputHash": "437e195fb1b0d4fd2450e8591dffc7ee4b9d6d8863f8ddf2b03f5fbb5385a594", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-5e119a10a0431405", + "sequence": 1, + "previousHash": "5b8d2a6ccfdb48ad39ba2fef170bdde7c1dc6f6a505449b015498cb4033ab513", + "rowHash": "5ddd515a66e2e8b06227aa679af5f7ea734ad4d500ef04a94e91e6cb5b344dcb" + } + ], + "rowCount": 2, + "chainHead": "5ddd515a66e2e8b06227aa679af5f7ea734ad4d500ef04a94e91e6cb5b344dcb", + "transcriptsPublished": false, + "endedAt": "2026-08-10T17:23:20.293Z", + "ledgerHash": "f5c2dac342edf319c7b9d63e0ecbbf04182799659b0577c134a181c75ec90df4", + "signature": "npYNHbBfAjS2N8AS6e7NzxRIKdMkDVH1B9QPlQqTJxQoC5/iBHMbPf4lPGCs8FSHZoOQhtz7SIeGzPVs+HC6Aw==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "f5c2dac342edf319c7b9d63e0ecbbf04182799659b0577c134a181c75ec90df4", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "f5c2dac342edf319c7b9d63e0ecbbf04182799659b0577c134a181c75ec90df4" + ], + "rejectedLedgers": 0, + "derivedHash": "ebb801e0ba9d90897964a026adbd876d32dfea22bba6b4bb286c1ce8fad8071c" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "7e2929e63f64a2981dd3804480dd1bbe6c771bc0e2747195eba7b489fce2286d" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v6-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v6-attempts.jsonl new file mode 100644 index 000000000..523484da2 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v6-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":[],"providerFailures":[],"rowHash":"83adadb130b15432706cef1fea6f5163a3f0c3b69eb7b0d07d30c8ac5cd1a239","runnerExitCode":0,"trialId":"claude-code->codex:federation-001:0:empty","valid":true,"validation":[]} +{"arm":"runtime","causalEvents":6,"driverValidation":[],"providerFailures":[],"rowHash":"8b917775a1f81046477391a8076e43c8bf47d5ee2477a2f8cfc5c1c7c6312a15","runnerExitCode":0,"trialId":"claude-code->codex:federation-001:0:runtime","valid":true,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v6.json b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v6.json new file mode 100644 index 000000000..4368b8f05 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v6.json @@ -0,0 +1,673 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T17:31:13.862Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook", + "transport": "codex-app-server" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook", + "transport": "claude-code" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "1.1.11", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook", + "transport": "antigravity" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": false, + "crossSession": true, + "crossProject": true, + "concurrentAgents": false, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.2857142857142857, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 2, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "claude-code->codex", + "families": null, + "arms": "empty,runtime", + "requestedArms": "empty,runtime", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "97b8d742145106766dcbddae8271634e5945a8553263d5bb124f6b2110eb7318" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": true, + "trialIntegrity": true, + "taskCorrect": false, + "runtimeDelivered": true, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": true + }, + "failed": [ + "taskCorrect" + ], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": -0.17122430244526665 + }, + "failures": [], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-dc9aeea268e7f81f", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "e5f0f6751893c915c957266789ad14969833a975adc0c8d0bf0b1b1f99258fca", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T17:31:13.857Z", + "runHash": "1773b5c1582dd85b393ab907574fb8c7f50bcdd628f526f21fba5a2c021fff3a" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "claude-code->codex:federation-001:0:empty", + "pairId": "claude-code->codex:federation-001:0", + "studySequence": 52, + "taskId": "federation-001", + "family": "cross-project-generalization", + "arm": "empty", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "cross-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "99f2ebb96e250c908ad7449f409a0ff35e1c6d3c5c1f853b26108fea1426c297", + "poweredStratum": false, + "correct": false, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 27452, + "totalTokens": 35947, + "latencyMs": 171295, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "workspaceStateHash": "8ec80f4bdc5b4b4b483ecc28b4536080c51c86547d810d603ef53c9b0ddcbc99", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "d7223a4e45d671f9b428b3e9333b70702ff4244e7ddaa00fce5b6388d9d6e170", + "runnerExitCode": 0, + "runnerOutputHash": "c0557be8d70173cd1fcd550ed3ea7367487f02ce30146526df61807ad8ceabd9", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-dc9aeea268e7f81f", + "sequence": 0, + "previousHash": "1773b5c1582dd85b393ab907574fb8c7f50bcdd628f526f21fba5a2c021fff3a", + "rowHash": "f64e6d858b37ac50e8469b6f62f36e52cbf1dd48acb37237836645062f4863e1" + }, + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "claude-code->codex:federation-001:0:runtime", + "pairId": "claude-code->codex:federation-001:0", + "studySequence": 551, + "taskId": "federation-001", + "family": "cross-project-generalization", + "arm": "runtime", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "cross-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "52c8a245c63f3a789640257921352cbcb66301faf0e652c34b2d35beedab82c0", + "poweredStratum": false, + "correct": false, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": true, + "deliveryPhase": "pre-action", + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0.018714244157859012, + "reconstructionTokens": 20626, + "totalTokens": 29792, + "latencyMs": 132444, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 386, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "workspaceStateHash": "8ec80f4bdc5b4b4b483ecc28b4536080c51c86547d810d603ef53c9b0ddcbc99", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "33a8405937ccec0fdd2b114c4d7ea61fae984aac2b7b977fd2318145b6122ba4", + "runnerExitCode": 0, + "runnerOutputHash": "5821be9d3addb529d15bede54e646e4bc4fc854877d91293ba9818354de99fac", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-dc9aeea268e7f81f", + "sequence": 1, + "previousHash": "f64e6d858b37ac50e8469b6f62f36e52cbf1dd48acb37237836645062f4863e1", + "rowHash": "b9a2ba277906505dc7b2f57559955ea7f506be73166deff7656058e2bfd6379c" + } + ], + "rowCount": 2, + "chainHead": "b9a2ba277906505dc7b2f57559955ea7f506be73166deff7656058e2bfd6379c", + "transcriptsPublished": false, + "endedAt": "2026-08-10T17:31:13.858Z", + "ledgerHash": "2a876004e74083afd9b572815de2f25d7275d8ece663a2ac334910de82e8a1a5", + "signature": "fjIH3rYmjNupLbkXPTkZc7wYqRNE6DfS5F4RMTAE+lNMKL6UphCLHQD3VkcQyYWmHOdauzo9dIWtQcxk/MpRCQ==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "2a876004e74083afd9b572815de2f25d7275d8ece663a2ac334910de82e8a1a5", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "2a876004e74083afd9b572815de2f25d7275d8ece663a2ac334910de82e8a1a5" + ], + "rejectedLedgers": 0, + "derivedHash": "f463c6be8c775adb8c37344134a03a9e17a962cebf1db411581c946ad9f3d7aa" + }, + "costDiagnostics": { + "passed": true, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [ + { + "dimension": "direction", + "value": "claude-code->codex", + "pairs": 1, + "tokenOverhead": -0.17122430244526665, + "latencyOverheadP95": -0.22680755421932922, + "regressions": [], + "passed": true + }, + { + "dimension": "family", + "value": "cross-project-generalization", + "pairs": 1, + "tokenOverhead": -0.17122430244526665, + "latencyOverheadP95": -0.22680755421932922, + "regressions": [], + "passed": true + }, + { + "dimension": "client", + "value": "codex", + "pairs": 1, + "tokenOverhead": -0.17122430244526665, + "latencyOverheadP95": -0.22680755421932922, + "regressions": [], + "passed": true + } + ] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "8ebbbdd6e5ced34c2e703c3027f2c982646b1f3cb3ea25fa59c2df72d43db17a" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v7-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v7-attempts.jsonl new file mode 100644 index 000000000..6afb860d2 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v7-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":[],"providerFailures":[],"rowHash":"54e589361a7b8e0a9b35e52064e7431e4895f10fdd385b167cd7f3d7195507cc","runnerExitCode":0,"trialId":"claude-code->codex:federation-001:0:empty","valid":true,"validation":[]} +{"arm":"runtime","causalEvents":6,"driverValidation":[],"providerFailures":[],"rowHash":"e06e9a086ceba4b8cde6f9e3f889ed3e32d86b84942e1973d1e6c0aa1aea9dcf","runnerExitCode":0,"trialId":"claude-code->codex:federation-001:0:runtime","valid":true,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v7.json b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v7.json new file mode 100644 index 000000000..66906d12b --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v7.json @@ -0,0 +1,671 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T17:47:01.780Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook", + "transport": "codex-app-server" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook", + "transport": "claude-code" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "1.1.11", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook", + "transport": "antigravity" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": false, + "crossSession": true, + "crossProject": true, + "concurrentAgents": false, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.2857142857142857, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 2, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "claude-code->codex", + "families": null, + "arms": "empty,runtime", + "requestedArms": "empty,runtime", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "97b8d742145106766dcbddae8271634e5945a8553263d5bb124f6b2110eb7318" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "passed", + "passed": true, + "checks": { + "selectionComplete": true, + "trialIntegrity": true, + "taskCorrect": true, + "runtimeDelivered": true, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": true + }, + "failed": [], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": -0.3253001751912148 + }, + "failures": [], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-c419e1d17eed8b61", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "e5f0f6751893c915c957266789ad14969833a975adc0c8d0bf0b1b1f99258fca", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T17:47:01.776Z", + "runHash": "534f9973ac49968631d7af27073cbae4e9ca2861151b5c7c035cedeced967ad9" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "claude-code->codex:federation-001:0:empty", + "pairId": "claude-code->codex:federation-001:0", + "studySequence": 52, + "taskId": "federation-001", + "family": "cross-project-generalization", + "arm": "empty", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "cross-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "99f2ebb96e250c908ad7449f409a0ff35e1c6d3c5c1f853b26108fea1426c297", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 14958, + "totalTokens": 23403, + "latencyMs": 114707, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "66d198d4d4f5d6152187c07fb5ad60801eed80e327cbfb237cfaeb46ce77539b", + "workspaceStateHash": "fb79ff90af607996ac49ad78f5edd6790837a6cb3959cf04a82174621fea0a57", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "d6108152315c9ebcb4947294c03ec425636c6c4b7b66d8eb6ee174c0a8dbf5b8", + "runnerExitCode": 0, + "runnerOutputHash": "280c8442e592c6e40fde9eed5b217641e2aecae48edf56d5292b87a12e5dc7e1", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-c419e1d17eed8b61", + "sequence": 0, + "previousHash": "534f9973ac49968631d7af27073cbae4e9ca2861151b5c7c035cedeced967ad9", + "rowHash": "0fbdd3381b5efe79cc991a1ea582f2eab652768f7c0e2fb69f7cc4294cc5cd0a" + }, + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "claude-code->codex:federation-001:0:runtime", + "pairId": "claude-code->codex:federation-001:0", + "studySequence": 551, + "taskId": "federation-001", + "family": "cross-project-generalization", + "arm": "runtime", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "cross-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "52c8a245c63f3a789640257921352cbcb66301faf0e652c34b2d35beedab82c0", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": true, + "deliveryPhase": "pre-action", + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0.02219579865239794, + "reconstructionTokens": 15138, + "totalTokens": 15790, + "latencyMs": 87670, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 336, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "e47eeba0dcb10be2e568adad6357383b3ce1928d867ecf58eb2ad555e74d6736", + "workspaceStateHash": "763750ae866950b9af0ed97670a581cd8a70e136e54cf4e1ae7fccdbd1d9cdd5", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "d18c6009fe80f210f96f71563da983090df09b6755e00200537b49155db9b6dd", + "runnerExitCode": 0, + "runnerOutputHash": "73bd9e4f64cce8cfa748f87881513ed1e24e35dc9d924d36619ffa08a8167abf", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-c419e1d17eed8b61", + "sequence": 1, + "previousHash": "0fbdd3381b5efe79cc991a1ea582f2eab652768f7c0e2fb69f7cc4294cc5cd0a", + "rowHash": "0388f2682496f05b927854d0e2d557cc120e72c1e26e009a6f9dcbe69ae7453a" + } + ], + "rowCount": 2, + "chainHead": "0388f2682496f05b927854d0e2d557cc120e72c1e26e009a6f9dcbe69ae7453a", + "transcriptsPublished": false, + "endedAt": "2026-08-10T17:47:01.777Z", + "ledgerHash": "8974e84b8d94637b672aa9f476cb836f317fec548409bf563c0780b4e70c14fc", + "signature": "rzcPAeAjallz5ia1l9dFNQcDUmDjLF7HciEPxe8McWvEc2T55eWqoT/55mxzNF9oMMdvdItV8armlCdhSoGlCg==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "8974e84b8d94637b672aa9f476cb836f317fec548409bf563c0780b4e70c14fc", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "8974e84b8d94637b672aa9f476cb836f317fec548409bf563c0780b4e70c14fc" + ], + "rejectedLedgers": 0, + "derivedHash": "2c78f8359ea8aa74fc6f8b55f20dc2ee77f437eaf18d543ff1be5cf92a5dddb2" + }, + "costDiagnostics": { + "passed": true, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [ + { + "dimension": "direction", + "value": "claude-code->codex", + "pairs": 1, + "tokenOverhead": -0.3253001751912148, + "latencyOverheadP95": -0.23570488287550018, + "regressions": [], + "passed": true + }, + { + "dimension": "family", + "value": "cross-project-generalization", + "pairs": 1, + "tokenOverhead": -0.3253001751912148, + "latencyOverheadP95": -0.23570488287550018, + "regressions": [], + "passed": true + }, + { + "dimension": "client", + "value": "codex", + "pairs": 1, + "tokenOverhead": -0.3253001751912148, + "latencyOverheadP95": -0.23570488287550018, + "regressions": [], + "passed": true + } + ] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "4870ae8cd66df4d4205ecbf977807eeba97c18ef1bb453ec71ea10e0e2de98fd" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v8-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v8-attempts.jsonl new file mode 100644 index 000000000..9fa342079 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v8-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":[],"providerFailures":[],"rowHash":"d5412c997fd560f2b9c8e4f0b96159b0329479dad1ada0e46ca118ea73a293a1","runnerExitCode":0,"trialId":"claude-code->codex:federation-001:0:empty","valid":true,"validation":[]} +{"arm":"runtime","causalEvents":6,"driverValidation":[],"providerFailures":[],"rowHash":"392ea3f962d88aff48e7de86f31f835685613c16f86f2bd6e024fbfcfc5302e6","runnerExitCode":0,"trialId":"claude-code->codex:federation-001:0:runtime","valid":true,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v8.json b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v8.json new file mode 100644 index 000000000..ce3fae113 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v8.json @@ -0,0 +1,768 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T17:57:09.357Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook", + "transport": "codex-app-server" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook", + "transport": "claude-code" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "1.1.11", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook", + "transport": "antigravity" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": false, + "crossSession": true, + "crossProject": true, + "concurrentAgents": false, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.2857142857142857, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 2, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "claude-code->codex", + "families": null, + "arms": "empty,runtime", + "requestedArms": "empty,runtime", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "97b8d742145106766dcbddae8271634e5945a8553263d5bb124f6b2110eb7318" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": true, + "trialIntegrity": true, + "taskCorrect": false, + "runtimeDelivered": true, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": false, + "tokenNonInferiority": false + }, + "failed": [ + "taskCorrect", + "correctnessNonInferiority", + "tokenNonInferiority" + ], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": 0.3299015019833311 + }, + "failures": [], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-052a55f49e9b3ca9", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "8075fd711aa542e58ac1b9e018b425747bbcb958a988e1eadab5cb1627658ce5", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T17:57:09.352Z", + "runHash": "b8e453a0ba723312a4391285935cdc8155ec2075d7c6857875487a298b311ac5" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "claude-code->codex:federation-001:0:empty", + "pairId": "claude-code->codex:federation-001:0", + "studySequence": 52, + "taskId": "federation-001", + "family": "cross-project-generalization", + "arm": "empty", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "producerModelVersion": "claude-sonnet-5", + "consumerModelVersion": "gpt-5.6-sol", + "producerTransport": "claude-code", + "consumerTransport": "codex-app-server", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "modelAttestations": [ + { + "role": "producer", + "transport": "claude-code", + "providerRequestId": "0669842b-b791-454d-b6e3-ea45bde68232", + "attestation": { + "attestationHash": "da35ed7b16d8d23acbc865e751f56740fed9eb57ed8ed1c1e77b20466fb983da", + "client": "claude-code", + "effectiveModel": "claude-sonnet-5", + "evidenceHash": "f9e0ef45cc3c95b04a483b5d8e9be113423e041c3396cf5647b7b6eb431611b3", + "provider": "anthropic", + "providerRequestId": "0669842b-b791-454d-b6e3-ea45bde68232", + "requestedModel": "claude-sonnet-5", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "claude-code/stream-json" + } + }, + { + "role": "consumer-1", + "transport": "codex-app-server", + "providerRequestId": "019feccf-1ff3-7fd2-91a9-b60b65fc8c83", + "attestation": { + "attestationHash": "1a094b8107e3106e34847db91f42dae799ac87f2ebea65691ce0e99b6141fa41", + "client": "codex", + "effectiveModel": "gpt-5.6-sol", + "evidenceHash": "3a5ad486c89b2c8d46bc0a766ef9bdbdb6d1761285b44ec05960f224291b6278", + "provider": "openai", + "providerRequestId": "019feccf-1ff3-7fd2-91a9-b60b65fc8c83", + "requestedModel": "gpt-5.6-sol", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "codex-app-server/thread-start" + } + } + ], + "consumerMcpExposed": false, + "actionAuditHash": "5ec64ecb7f6ce0777d1498dd26768bf85faa7e5683f1399cc1f3ce6241772507", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "cross-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "99f2ebb96e250c908ad7449f409a0ff35e1c6d3c5c1f853b26108fea1426c297", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 15116, + "totalTokens": 22437, + "latencyMs": 94172, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "d8ca39fa73320c5eb5451e0cf35d1d482fc14d245cf43b0749b77809c407b1be", + "workspaceStateHash": "1d9af69553d682309898ab65fcb18c9ed206daf6bc035aca47dd18a6c43d88f5", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "a7240bc8e7978f4b803e63950513c8c1653d360f319b9d641b79c2fd8c4086de", + "runnerExitCode": 0, + "runnerOutputHash": "df140c165113a29f2eefbef0206f77a1053aee5288b183f824896dbee14fa731", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-052a55f49e9b3ca9", + "sequence": 0, + "previousHash": "b8e453a0ba723312a4391285935cdc8155ec2075d7c6857875487a298b311ac5", + "rowHash": "857488602c8be013d06184df6ce4a21afc0b06828c33cb48de7710bd71a1b10a" + }, + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "claude-code->codex:federation-001:0:runtime", + "pairId": "claude-code->codex:federation-001:0", + "studySequence": 551, + "taskId": "federation-001", + "family": "cross-project-generalization", + "arm": "runtime", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "producerModelVersion": "claude-sonnet-5", + "consumerModelVersion": "gpt-5.6-sol", + "producerTransport": "claude-code", + "consumerTransport": "codex-app-server", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "modelAttestations": [ + { + "role": "producer", + "transport": "claude-code", + "providerRequestId": "dd5594dc-f2a4-4e84-83d3-eccd90f01928", + "attestation": { + "attestationHash": "b8eca2a5a141249656dc1cb7ad383df61fa0e9b620762c36da1812f924b50b08", + "client": "claude-code", + "effectiveModel": "claude-sonnet-5", + "evidenceHash": "93e87e7efe0d7e9469cd196117071d4d16667b501a41b77983b901678d46c03f", + "provider": "anthropic", + "providerRequestId": "dd5594dc-f2a4-4e84-83d3-eccd90f01928", + "requestedModel": "claude-sonnet-5", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "claude-code/stream-json" + } + }, + { + "role": "consumer-1", + "transport": "codex-app-server", + "providerRequestId": "019fecd1-b418-7ea0-8cd2-b8fddbd57dd6", + "attestation": { + "attestationHash": "98ab24439c342ff668ec6945de8cfef5174345e09d834d1e7b64588f50ec7b02", + "client": "codex", + "effectiveModel": "gpt-5.6-sol", + "evidenceHash": "9c66189c8a8aca03457b61222f98dcf04c58d0f9ad4fea739b801b733fbb0abb", + "provider": "openai", + "providerRequestId": "019fecd1-b418-7ea0-8cd2-b8fddbd57dd6", + "requestedModel": "gpt-5.6-sol", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "codex-app-server/thread-start" + } + } + ], + "consumerMcpExposed": false, + "actionAuditHash": "a1aedd0128e619c108b740823e43b98ce3e4e8fe7411a74b9f257810523aaf22", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "cross-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "52c8a245c63f3a789640257921352cbcb66301faf0e652c34b2d35beedab82c0", + "poweredStratum": false, + "correct": false, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": true, + "deliveryPhase": "pre-action", + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0.027526990340299262, + "reconstructionTokens": 15839, + "totalTokens": 29839, + "latencyMs": 176711, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 436, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "497a0708d15c357b29b8360ea82472c9785999f2119eea8f064af4a68887d927", + "workspaceStateHash": "f046898280ec4ca302ab404f221427742cff616c87816a5a0975417f924d21c1", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "142f610cf293623797afe42883e363437420dca03c1d46ebd2437043def8fe10", + "runnerExitCode": 0, + "runnerOutputHash": "6db6c860a80a803c53259347c59ab3f1bc70cb023f70fb0ce03ee33716d933d6", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-052a55f49e9b3ca9", + "sequence": 1, + "previousHash": "857488602c8be013d06184df6ce4a21afc0b06828c33cb48de7710bd71a1b10a", + "rowHash": "bdf64aa1b8dd56107d7b83efab94d0a324072f11f51bf24acd4afa071818226c" + } + ], + "rowCount": 2, + "chainHead": "bdf64aa1b8dd56107d7b83efab94d0a324072f11f51bf24acd4afa071818226c", + "transcriptsPublished": false, + "endedAt": "2026-08-10T17:57:09.353Z", + "ledgerHash": "aa5b285403df90b1350541ea89f6aad6c8a7d0aaccef3e242e76e10f9642dfb9", + "signature": "vZJ8mOcC+lBKOIX07WIhtQmeTRp9xV2Z3XHtIOKpvnCUnTl58ARq5tn5Jc6/v7OWfARpWZps99SYlu4g1L59Dw==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "aa5b285403df90b1350541ea89f6aad6c8a7d0aaccef3e242e76e10f9642dfb9", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "aa5b285403df90b1350541ea89f6aad6c8a7d0aaccef3e242e76e10f9642dfb9" + ], + "rejectedLedgers": 0, + "derivedHash": "ecbaa4cd6136e5578bedd7230f524c22fcbd3edfd46abd8e7f6347cdd6ad961a" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [ + { + "dimension": "direction", + "value": "claude-code->codex", + "pairs": 1, + "tokenOverhead": 0.3299015019833311, + "latencyOverheadP95": 0.8764707131631483, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + }, + { + "dimension": "family", + "value": "cross-project-generalization", + "pairs": 1, + "tokenOverhead": 0.3299015019833311, + "latencyOverheadP95": 0.8764707131631483, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + }, + { + "dimension": "client", + "value": "codex", + "pairs": 1, + "tokenOverhead": 0.3299015019833311, + "latencyOverheadP95": 0.8764707131631483, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + } + ] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "98e587d3979c694500a5def4a86804c05515d4a1567269e144e383c2ee73d554" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v9-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v9-attempts.jsonl new file mode 100644 index 000000000..625892fb3 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v9-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":[],"providerFailures":[],"rowHash":"98e6a048fba36105aa702852d86506c3daf470a8518c2979e1c3856f9c2cee76","runnerExitCode":0,"trialId":"claude-code->codex:federation-001:0:empty","valid":true,"validation":[]} +{"arm":"runtime","causalEvents":0,"driverValidation":["active model did not author semantic capture","model-authored semantic delta is incomplete","semantic capture lacks verified repository evidence","host action audit is incomplete"],"providerFailures":[{"classification":"provider-quota-exhausted","role":"producer"}],"rowHash":"45c016d3fc1b5ff272db447f6638de18f2433499819f84228384e50ab519feb0","runnerExitCode":1,"trialId":"claude-code->codex:federation-001:0:runtime","valid":false,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v9.json b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v9.json new file mode 100644 index 000000000..32f8b353e --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-claude-to-codex-v9.json @@ -0,0 +1,810 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T18:08:03.961Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook", + "transport": "codex-app-server" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook", + "transport": "claude-code" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "1.1.11", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook", + "transport": "antigravity" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": false, + "crossSession": true, + "crossProject": true, + "concurrentAgents": false, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.2857142857142857, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 2, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "claude-code->codex", + "families": null, + "arms": "empty,runtime", + "requestedArms": "empty,runtime", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "97b8d742145106766dcbddae8271634e5945a8553263d5bb124f6b2110eb7318" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": false, + "trialIntegrity": false, + "taskCorrect": true, + "runtimeDelivered": false, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": true + }, + "failed": [ + "selectionComplete", + "trialIntegrity", + "runtimeDelivered" + ], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": 0.029527692830445124 + }, + "failures": [ + { + "trialId": "claude-code->codex:federation-001:0:runtime", + "arm": "runtime", + "runnerExitCode": 1, + "valid": false, + "validation": [], + "driverValidation": [ + "active model did not author semantic capture", + "model-authored semantic delta is incomplete", + "semantic capture lacks verified repository evidence", + "host action audit is incomplete" + ], + "causalEvents": 0, + "providerFailures": [ + { + "classification": "provider-quota-exhausted", + "role": "producer" + } + ], + "rowHash": "45c016d3fc1b5ff272db447f6638de18f2433499819f84228384e50ab519feb0" + } + ], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-2f5b0818fb710ace", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "afabc04f3c39488063c546d3830006e8100118b928fb11ef7e033965cfb7dfbd", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T18:08:03.957Z", + "runHash": "806c0193e5c0de9977562845b7492146c4e4fa419e15ab13fa7c7ac2dfe3d4f7" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "claude-code->codex:federation-001:0:empty", + "pairId": "claude-code->codex:federation-001:0", + "studySequence": 52, + "taskId": "federation-001", + "family": "cross-project-generalization", + "arm": "empty", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "producerModelVersion": "claude-sonnet-5", + "consumerModelVersion": "gpt-5.6-sol", + "producerTransport": "claude-code", + "consumerTransport": "codex-app-server", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "modelAttestations": [ + { + "role": "producer", + "transport": "claude-code", + "providerRequestId": "326f6514-086a-4379-8063-5e725dd19af8", + "attestation": { + "attestationHash": "9e52cf8ee5ac13e695c19293c659efe73302c1234fea0eb5792fb977bc248d6e", + "client": "claude-code", + "effectiveModel": "claude-sonnet-5", + "evidenceHash": "63f782c28c3005147a73fbc418d4c3e7725f1e248f4a60df59cacef82fd957ed", + "provider": "anthropic", + "providerRequestId": "326f6514-086a-4379-8063-5e725dd19af8", + "requestedModel": "claude-sonnet-5", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "claude-code/stream-json" + } + }, + { + "role": "consumer-1", + "transport": "codex-app-server", + "providerRequestId": "019fecd8-b388-7842-a03f-735fdecf7baa", + "attestation": { + "attestationHash": "38d990d03f9d3fee50b3ad17d6553022992753480918c5a50ae52d1b3402da78", + "client": "codex", + "effectiveModel": "gpt-5.6-sol", + "evidenceHash": "2a534e9f27735e9064187cc9f612f2b378804555125a754f34884c8dd1700a3b", + "provider": "openai", + "providerRequestId": "019fecd8-b388-7842-a03f-735fdecf7baa", + "requestedModel": "gpt-5.6-sol", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "codex-app-server/thread-start" + } + } + ], + "consumerMcpExposed": false, + "actionAuditHash": "e441aa06567de869de3bfadc5cf863d21f65c6bb2a39e479d241a72fb036e9a0", + "semanticEvidenceVerification": { + "anchors": [ + { + "bytes": 97, + "path": "config/active-decision.json", + "sha256": "c6b1ba55ef77185f305a210e44e54362ab2f56436cc4dd715f84da9588f54fed" + }, + { + "bytes": 210, + "path": "config/supersession-log.json", + "sha256": "66874921ba5a911dcbb2f5326e70f40b9e54005793a3b083ddd0145dca474964" + }, + { + "bytes": 143, + "path": "federation/authorization.json", + "sha256": "243b1aa34d211415789710dd5d663d55660357ae94fb3a0f0bade78d71f87d81" + }, + { + "bytes": 87, + "path": "distractors/candidate-1.json", + "sha256": "6e1489b55ac0e5c3c69a1c23ab4ca6d5145f70e69c9eef0b23d6ff9d1f8a1056" + }, + { + "bytes": 87, + "path": "distractors/candidate-2.json", + "sha256": "d6f552455d4ba7f6b98aced0305980a2174a25ebbc617067161afdde4382571a" + }, + { + "bytes": 87, + "path": "distractors/candidate-3.json", + "sha256": "0998d47b6823d66381d22f12ce7641969ab831eb33b29326f732a0b5c037639d" + }, + { + "bytes": 87, + "path": "distractors/candidate-4.json", + "sha256": "db1fb4e6315944f517c5b6fb4b84d56ed11b04e29417dcc62b4a1f679925c996" + } + ], + "diagnostics": [], + "schemaVersion": "ucr.semantic-evidence-verification/1", + "verificationHash": "45149906b1be22ea5c2bfa2ab65dff99ea39d4dafadde9b33810f1a02eab1ad2", + "verified": true + }, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "cross-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "99f2ebb96e250c908ad7449f409a0ff35e1c6d3c5c1f853b26108fea1426c297", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 14962, + "totalTokens": 29430, + "latencyMs": 162893, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "3bcda82a5f1e1b0ec95a4ca02d7ab3f81b3c451de568acf656cd0b31d85e2b66", + "workspaceStateHash": "5f5bb91b45f093c8b399b84d169c393e01468fb1578d110f57dd6eb0f101f6b6", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "2084d44991537c05d07987ac0fe84b049418e81a53b31429fc58233d3e289706", + "runnerExitCode": 0, + "runnerOutputHash": "712c7068bf6589e0f611c6e0a8dc3205f720d51130ed6e585e43e079986edb65", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-2f5b0818fb710ace", + "sequence": 0, + "previousHash": "806c0193e5c0de9977562845b7492146c4e4fa419e15ab13fa7c7ac2dfe3d4f7", + "rowHash": "30947676853872d4408f9a8bf148dac314ce165b1f1d90eb5d3dd5d44eaa7069" + }, + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "claude-code->codex:federation-001:0:runtime", + "pairId": "claude-code->codex:federation-001:0", + "studySequence": 551, + "taskId": "federation-001", + "family": "cross-project-generalization", + "arm": "runtime", + "direction": "claude-code->codex", + "producerClient": "claude-code", + "consumerClient": "codex", + "producerFamily": "anthropic", + "consumerFamily": "openai", + "producerModel": "claude-sonnet-5", + "consumerModel": "gpt-5.6-sol", + "producerModelVersion": "claude-sonnet-5", + "consumerModelVersion": "gpt-5.6-sol", + "producerTransport": "claude-code", + "consumerTransport": "codex-app-server", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "modelAttestations": [ + { + "role": "producer", + "transport": "claude-code", + "providerRequestId": "b2d2fad1-0d04-4d8d-acb8-84c9f2fbf95c", + "attestation": { + "attestationHash": "59802493fa593d5419d81f5abd3211e55922e4faaeacdab0e59cb95caa975daf", + "client": "claude-code", + "effectiveModel": "claude-sonnet-5", + "evidenceHash": "54181b15033cf002ad7e7be5ef273723d2f1b7a5ea97413d8fdf118e58cb83de", + "provider": "anthropic", + "providerRequestId": "b2d2fad1-0d04-4d8d-acb8-84c9f2fbf95c", + "requestedModel": "claude-sonnet-5", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "claude-code/stream-json" + } + }, + { + "role": "consumer-1", + "transport": "codex-app-server", + "providerRequestId": "019fecdb-e478-7202-b41a-fa5b98f5d473", + "attestation": { + "attestationHash": "e43e40ef8bba7b1f6030d12e1e6e07d376bcf7c1df0c8fff15f3f8849aa170d1", + "client": "codex", + "effectiveModel": "gpt-5.6-sol", + "evidenceHash": "594b35afb57ca7eb0332ba6f3581d7112ee4c03e8ab8a211ec8acf81abbe0d18", + "provider": "openai", + "providerRequestId": "019fecdb-e478-7202-b41a-fa5b98f5d473", + "requestedModel": "gpt-5.6-sol", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "codex-app-server/thread-start" + } + } + ], + "consumerMcpExposed": false, + "actionAuditHash": "0a5e677943115fba6e9cac9f235a332dfbed5a89f68a94bcad5011d974d7690c", + "semanticEvidenceVerification": { + "anchors": [], + "diagnostics": [ + "capture has no verifiable repository evidence anchor" + ], + "schemaVersion": "ucr.semantic-evidence-verification/1", + "verificationHash": "214c955116e8c2670fb0df85a2a75c54443ebe78e9689eaad5e96aafdb9a6080", + "verified": false + }, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "cross-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "52c8a245c63f3a789640257921352cbcb66301faf0e652c34b2d35beedab82c0", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 14965, + "totalTokens": 30299, + "latencyMs": 204321, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [ + { + "classification": "provider-quota-exhausted", + "role": "producer" + } + ], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "e0a5036e6540968cc8a425d22dd6c0f00c0215351814ab68fc8c6610855788fe", + "workspaceStateHash": "789de10008f94ad4e6ea3dcae0b19cfac04009113699822f830989753eed2596", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "0e5a10bd37f8b1e500be00f715cc6fa807b0329f0298bc37336dd829d9699c40", + "runnerExitCode": 1, + "runnerOutputHash": "23aa4bf15212efa11dc3216e94013d52739e6ee9f669de6e1d6052e2642e3d7f", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-2f5b0818fb710ace", + "sequence": 1, + "previousHash": "30947676853872d4408f9a8bf148dac314ce165b1f1d90eb5d3dd5d44eaa7069", + "rowHash": "45ad862fc7cfb2a901fae95aa9aa697cc5a2e314b39a684e7304b84f0bd40e15" + } + ], + "rowCount": 2, + "chainHead": "45ad862fc7cfb2a901fae95aa9aa697cc5a2e314b39a684e7304b84f0bd40e15", + "transcriptsPublished": false, + "endedAt": "2026-08-10T18:08:03.957Z", + "ledgerHash": "3bcab5d71a25ddfc9ee4f6be4ca54acc47b8298300c412795cc71283d64da563", + "signature": "ffKNaH3tJ70axTQgfElf2Ul36/60Ipj+0FWCp4yAOCO/fU4x1at4sVktTQRV8dqfNyX9PR6jK4ZyXgKZxwT1BQ==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "3bcab5d71a25ddfc9ee4f6be4ca54acc47b8298300c412795cc71283d64da563", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "3bcab5d71a25ddfc9ee4f6be4ca54acc47b8298300c412795cc71283d64da563" + ], + "rejectedLedgers": 0, + "derivedHash": "44dc5741baeeaf687a12151ba3be212466f63be2c496d933fa6e8b2f9604aeeb" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "7e4a9858ef6d0d69d4014900a679417e2e3c53eb7d043b159625bd120aa4ee83" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v1-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v1-attempts.jsonl new file mode 100644 index 000000000..0463a86b9 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v1-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":[],"rowHash":"f8ff36c3e4b3bf74d348f5f992291bd670f7a4611957435069ca211844dd27aa","runnerExitCode":0,"trialId":"codex->claude-code:cross-model-001:0:empty","valid":true,"validation":[]} +{"arm":"runtime","causalEvents":6,"driverValidation":[],"rowHash":"0bc003de5e34832db0bd2e144fbfdb8c891b385f15d75d367e71f9b0ff09de77","runnerExitCode":0,"trialId":"codex->claude-code:cross-model-001:0:runtime","valid":true,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v1.json b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v1.json new file mode 100644 index 000000000..b3a116707 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v1.json @@ -0,0 +1,608 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T14:23:46.586Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "gemini-cli 0.28.1", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "304663a636671cd838304ae46a22aca0d42685f543ea17d391d1574fbf952f71", + "trialCount": 693 + }, + "coverage": { + "passed": true, + "checks": { + "effectivenessDesign": true, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [] + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "failures": [], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-2e7403b40bf6d521", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "6d56b0f65c183dad638e2dbd2c4486281dbbc90567c42953618f060e10e1ea0a", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "304663a636671cd838304ae46a22aca0d42685f543ea17d391d1574fbf952f71", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T14:23:46.581Z", + "runHash": "a223fedeb616034e67c2f2ac001c978b006b8631165ac7f74620bfc1cbc28f00" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "304663a636671cd838304ae46a22aca0d42685f543ea17d391d1574fbf952f71", + "trialId": "codex->claude-code:cross-model-001:0:empty", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 37, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "empty", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "4f5777767e446baf7e658dd317134ebde2f555950387b8ac1a67e66459a5aeb8", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 2, + "totalTokens": 43791, + "latencyMs": 80016, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "42ea57ca5a1632573b2b1bb87486d40095a63fe0dfd6dae32f26ae1d9731a49d", + "workspaceStateHash": "d514523ad74921a7687a7aec2a96f45aef2def87b26f96e09d1ea4bb53ea7952", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "dc2311a0db52f6ed02ff5d92bce43f590191e1ab50e8db9dc357266430a2bfb8", + "runnerExitCode": 0, + "runnerOutputHash": "2c9a0aab70f544f3d93e61087e292f593c9ad1584b816d782ebfcd4983b3996b", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-2e7403b40bf6d521", + "sequence": 0, + "previousHash": "a223fedeb616034e67c2f2ac001c978b006b8631165ac7f74620bfc1cbc28f00", + "rowHash": "dbf109e9462fb0b1c3ec2e9e9551ddb3165aec3306fbeb0c16c51ba34bcc99a4" + }, + { + "study": "full-effectiveness", + "planHash": "304663a636671cd838304ae46a22aca0d42685f543ea17d391d1574fbf952f71", + "trialId": "codex->claude-code:cross-model-001:0:runtime", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 340, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "runtime", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "90bc20152c4ca188e4703921bb2a20cb5dddf276e8b99f98ca953604448f538a", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": true, + "deliveryPhase": "pre-action", + "stale": false, + "contradictory": false, + "contextOverheadRatio": 147, + "reconstructionTokens": 2, + "totalTokens": 62003, + "latencyMs": 106034, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 294, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "a39c75a8d4f372de758b4545ed3f1c9610ce0fea7cfbdfcf060665fc886f583b", + "workspaceStateHash": "6e2c3729c061c8f34d1a58bc510c8844e22fa8829b12ae1af33d8743ec21d931", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "780a823071198c86498cb41c575cbce225d737da927863eecdacd7ca8b4f765a", + "runnerExitCode": 0, + "runnerOutputHash": "94d22dd263ee454fde043df6a96b005d7e70f6501b77705c16c509991717d381", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-2e7403b40bf6d521", + "sequence": 1, + "previousHash": "dbf109e9462fb0b1c3ec2e9e9551ddb3165aec3306fbeb0c16c51ba34bcc99a4", + "rowHash": "7c8e5e755223cb15b9b924b8b2b81ba6877d3a42cbb5b5d122c183f54c5f025a" + } + ], + "rowCount": 2, + "chainHead": "7c8e5e755223cb15b9b924b8b2b81ba6877d3a42cbb5b5d122c183f54c5f025a", + "transcriptsPublished": false, + "endedAt": "2026-08-10T14:23:46.582Z", + "ledgerHash": "78c7e808213199debba047880be32b6ac1c7647bf72a23057b9ba8d2fd3304b1", + "signature": "Heb8JRlZqE5zAn6SdcW0kiRpjSF+oQR00ahrJbafYiJq5Nrwgyj4sie5DBsEo4tXXIJQceEvb0laAeHgkqMnDg==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "78c7e808213199debba047880be32b6ac1c7647bf72a23057b9ba8d2fd3304b1", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "78c7e808213199debba047880be32b6ac1c7647bf72a23057b9ba8d2fd3304b1" + ], + "rejectedLedgers": 0, + "derivedHash": "a2aaea46ba70a41862db4c3097c48606e5519f91fc2b2b809e4101c0491beba7" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [ + { + "dimension": "direction", + "value": "codex->claude-code", + "pairs": 1, + "tokenOverhead": 0.41588454248589896, + "latencyOverheadP95": 0.32515996800639874, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + }, + { + "dimension": "family", + "value": "cross-model-handoff", + "pairs": 1, + "tokenOverhead": 0.41588454248589896, + "latencyOverheadP95": 0.32515996800639874, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + }, + { + "dimension": "client", + "value": "claude-code", + "pairs": 1, + "tokenOverhead": 0.41588454248589896, + "latencyOverheadP95": 0.32515996800639874, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + } + ] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "a1019257d1236130312e568d45b8e382fe29d060663c8ec2d5946339de87880f" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v2-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v2-attempts.jsonl new file mode 100644 index 000000000..eb96ed115 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v2-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":[],"rowHash":"88282019cb81d6aca843703d0cb243fa6186935748854ba67b517c7eaace859b","runnerExitCode":0,"trialId":"codex->claude-code:cross-model-001:0:empty","valid":true,"validation":[]} +{"arm":"runtime","causalEvents":6,"driverValidation":[],"rowHash":"49a133803c842fe6062a681216742270679a2fe1d26de56fc8ab1daf616d94e4","runnerExitCode":0,"trialId":"codex->claude-code:cross-model-001:0:runtime","valid":true,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v2.json b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v2.json new file mode 100644 index 000000000..54dddb1e1 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v2.json @@ -0,0 +1,605 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T14:54:24.099Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "gemini-cli 0.28.1", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialCount": 693 + }, + "coverage": { + "passed": true, + "checks": { + "effectivenessDesign": true, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [] + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "failures": [], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-52538fbf4906eaac", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "73f83b7752a494dc71d2c2d626b12e9922c7dbdc2f6f879f16ebb29b23bca5ab", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T14:54:24.094Z", + "runHash": "59fa8a7873938dec8649f33d94016bb8375bd58beca06cb35959df337a83eb35" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "codex->claude-code:cross-model-001:0:empty", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 37, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "empty", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "4f5777767e446baf7e658dd317134ebde2f555950387b8ac1a67e66459a5aeb8", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 2, + "totalTokens": 43406, + "latencyMs": 52268, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "4c2201a9c0803014ddbf184c166346ae33eba38dfa27b8c1baf37f3d963d6473", + "workspaceStateHash": "18f947e182e1322e8bf253c5dec168c4004627f48da2059f16e188b526ed8a7b", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "086393eda728218852b32af603f9f7a272b55258ebade49903f30032fc6c924d", + "runnerExitCode": 0, + "runnerOutputHash": "d54525dd86015204fd23c31bbbe4731a32781f10ebef256825491117e3a36485", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-52538fbf4906eaac", + "sequence": 0, + "previousHash": "59fa8a7873938dec8649f33d94016bb8375bd58beca06cb35959df337a83eb35", + "rowHash": "039116de82a5e6e3db5970af90cf206fa5a2bfc7334ca860b493ea645146334a" + }, + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "codex->claude-code:cross-model-001:0:runtime", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 340, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "runtime", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "90bc20152c4ca188e4703921bb2a20cb5dddf276e8b99f98ca953604448f538a", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": true, + "deliveryPhase": "pre-action", + "stale": false, + "contradictory": false, + "contextOverheadRatio": 113, + "reconstructionTokens": 2, + "totalTokens": 43434, + "latencyMs": 58410, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 226, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "106b54ecb3284a495d2226dd60d0f62fcf52587cac5eae35fc702c2d65796ef1", + "workspaceStateHash": "1242c720484a99f5edcb41d8315bc57410bec278e455b90fb3b9fc90dbef0505", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "c032fe82150f8b71d6d2a287c37c8cb262fbb0e2e6d85f3b336278af74c6eafa", + "runnerExitCode": 0, + "runnerOutputHash": "4d651d27e91d440a3a6c98911866cebddee0765374774bbbaaa7ba3875ab94ae", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-52538fbf4906eaac", + "sequence": 1, + "previousHash": "039116de82a5e6e3db5970af90cf206fa5a2bfc7334ca860b493ea645146334a", + "rowHash": "44d54c87248fa495db458c9a856fbf50e74a19bfcdcc00301481b9b27f46bb25" + } + ], + "rowCount": 2, + "chainHead": "44d54c87248fa495db458c9a856fbf50e74a19bfcdcc00301481b9b27f46bb25", + "transcriptsPublished": false, + "endedAt": "2026-08-10T14:54:24.095Z", + "ledgerHash": "235d0b546f92f312450a24531b37cb0be5a0f512855858d083efd870ca12f1ad", + "signature": "Vb45jMkHld0JGCWcKAy8LLLHYMGP3KR2QHfJ+qlq/Ik6MMgWW350hdtmwxK29BxXouXH71DRqT9ejXjKo8/gAQ==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "235d0b546f92f312450a24531b37cb0be5a0f512855858d083efd870ca12f1ad", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "235d0b546f92f312450a24531b37cb0be5a0f512855858d083efd870ca12f1ad" + ], + "rejectedLedgers": 0, + "derivedHash": "a2aaea46ba70a41862db4c3097c48606e5519f91fc2b2b809e4101c0491beba7" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [ + { + "dimension": "direction", + "value": "codex->claude-code", + "pairs": 1, + "tokenOverhead": 0.000645072109846565, + "latencyOverheadP95": 0.11750975740414785, + "regressions": [ + "latency" + ], + "passed": false + }, + { + "dimension": "family", + "value": "cross-model-handoff", + "pairs": 1, + "tokenOverhead": 0.000645072109846565, + "latencyOverheadP95": 0.11750975740414785, + "regressions": [ + "latency" + ], + "passed": false + }, + { + "dimension": "client", + "value": "claude-code", + "pairs": 1, + "tokenOverhead": 0.000645072109846565, + "latencyOverheadP95": 0.11750975740414785, + "regressions": [ + "latency" + ], + "passed": false + } + ] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "91ec6ef245f2a5d583589ab65e9990047e8e7774a97dcf97e50eb993b20cf8c1" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v3-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v3-attempts.jsonl new file mode 100644 index 000000000..651d0a92b --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v3-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":[],"rowHash":"01e5ada4d9e7558e15b2dd5537310f8b317540080a1ae43e2fbb55cad33abd62","runnerExitCode":0,"trialId":"codex->claude-code:cross-model-001:0:empty","valid":true,"validation":[]} +{"arm":"runtime","causalEvents":6,"driverValidation":[],"rowHash":"1044976c9ab82d75c2d6bbcef17a454ece8598ad42648bd6cb191c3c35a889c6","runnerExitCode":0,"trialId":"codex->claude-code:cross-model-001:0:runtime","valid":true,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v3.json b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v3.json new file mode 100644 index 000000000..822bd4b7b --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v3.json @@ -0,0 +1,805 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T14:59:33.763Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "gemini-cli 0.28.1", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialCount": 693 + }, + "coverage": { + "passed": true, + "checks": { + "effectivenessDesign": true, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [] + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "codex->claude-code", + "families": "cross-model-handoff", + "arms": "runtime,empty", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "6f7f5e0bd8a59e11f69978a7c36109a50dbaa42854bef10f313a6263be46e921" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": true, + "trialIntegrity": true, + "taskCorrect": false, + "runtimeDelivered": true, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": true + }, + "failed": [ + "taskCorrect" + ], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": -0.26170897390060777 + }, + "failures": [], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-55446bee60219f2c", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "04b94602e8f05cf10b372097f7a937202ac4d56d4e3d2369cad98a926e000497", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T14:59:33.758Z", + "runHash": "fef914b744683eb9b551ad09e8702b6672e0536357b122d1b8eaaf4d7c0f6130" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "codex->claude-code:cross-model-001:0:empty", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 37, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "empty", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "4f5777767e446baf7e658dd317134ebde2f555950387b8ac1a67e66459a5aeb8", + "poweredStratum": false, + "correct": false, + "severeHarm": false, + "mistakeExecuted": true, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 2, + "totalTokens": 58737, + "latencyMs": 72374, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "d2822f3c62a3b22fa324dbd3d7b10f8d7c4d22514fd761c25bc6054f484a3f38", + "workspaceStateHash": "0505d455869a0272ef551f36360ac1c459a62b4014013af9a8fc84e826fda130", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "48c07c84f79674e6d2e46c89f18bd2117ff9d8975e5d932eb2033a34a40ba3ba", + "runnerExitCode": 0, + "runnerOutputHash": "987a6fbd0e84c6a6f19619ae2c3fb8d77fe463c2230c99ba8ff36cf72d23f9e6", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-55446bee60219f2c", + "sequence": 0, + "previousHash": "fef914b744683eb9b551ad09e8702b6672e0536357b122d1b8eaaf4d7c0f6130", + "rowHash": "48fcb9562a387c34ce971096043a0c7de5129d14bc051e55f194f5b129075bd8" + }, + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "codex->claude-code:cross-model-001:0:runtime", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 340, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "runtime", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "90bc20152c4ca188e4703921bb2a20cb5dddf276e8b99f98ca953604448f538a", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": true, + "deliveryPhase": "pre-action", + "stale": false, + "contradictory": false, + "contextOverheadRatio": 116, + "reconstructionTokens": 2, + "totalTokens": 43365, + "latencyMs": 73836, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 232, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "causalClaim": true, + "causalChain": { + "stages": [ + { + "stage": "captured", + "eventId": "causal:6daa1c387196cdd765473ab4", + "parentEventId": null, + "artifactHash": "b9abb5753bf20e82e78116646f1fe03e143fd24fc114112fff0173cae99bcd85" + }, + { + "stage": "verified", + "eventId": "causal:1d7adec21e9f279b7f58d571", + "parentEventId": "causal:6daa1c387196cdd765473ab4", + "artifactHash": "7ab4d524afbc3e211b263af61fbb99d4f6325482329ad5a0b777055710e9b700" + }, + { + "stage": "eligible", + "eventId": "causal:6cd986bc4290c089b35004f7", + "parentEventId": "causal:1d7adec21e9f279b7f58d571", + "artifactHash": "4a3e71fcc5e0ac4a3b2ab4ec9e6276e566323ec7945118e04f10dd26a442e6e4" + }, + { + "stage": "retrieved", + "eventId": "causal:82f605cb8f84379d5d04ef47", + "parentEventId": "causal:6cd986bc4290c089b35004f7", + "artifactHash": "b0cdfe1be9e0daef36f6319a0e0e6b59665818d68caa7c6b7265c50eae4a4793" + }, + { + "stage": "delivered", + "eventId": "causal:4edf1cbd6fd24de0da380ea9", + "parentEventId": "causal:82f605cb8f84379d5d04ef47", + "artifactHash": "ea9fa8ecd980c0e8f5218f0cd9f0a6685d33bfb42d26b18e503a2246e3318c39" + }, + { + "stage": "used", + "eventId": "causal:0aa7c455904ee38a09667ad5", + "parentEventId": "causal:4edf1cbd6fd24de0da380ea9", + "artifactHash": "2481c0a7b32d87412d54d922455cae08d4cd531f84ff3299dbb9102089a8602c" + }, + { + "stage": "behaviorChanged", + "eventId": "causal:8558badde8a3cac43d2e6cd5", + "parentEventId": "causal:0aa7c455904ee38a09667ad5", + "artifactHash": "dcafdac407062be6169439feac55c35054251bfb4041df32aeec3055fe54b688" + }, + { + "stage": "mistakePrevented", + "eventId": "causal:57c71807583938453dcb1146", + "parentEventId": "causal:8558badde8a3cac43d2e6cd5", + "artifactHash": "6e2aa189dfd32ffee6fede58adeef7f5a25b9a8cbce27bdf0579a9c462b1b164" + }, + { + "stage": "taskCorrect", + "eventId": "causal:72ff837c93d3a3a74cdbf63b", + "parentEventId": "causal:57c71807583938453dcb1146", + "artifactHash": "d716653fd0d24ed5cae5eb878474dd6297458b45ae6e95cf3f24e9a21e74e32d" + } + ], + "events": [ + { + "eventId": "causal:6daa1c387196cdd765473ab4", + "stage": "captured", + "parentEventId": null, + "observer": "host", + "observedAt": 1, + "artifact": { + "deliveryHash": "fe7a64c4b079e5051ee322c3e14837fdd63480fbd3bfafa1150dae5d820e6a86", + "semanticDeltaHash": "38a6faf4120d6ff5418c07c74f17d7c1d4b6c5cb98b8a91d6fa0e5381de56e71", + "stage": "captured", + "trialId": "codex->claude-code:cross-model-001:0:runtime" + } + }, + { + "eventId": "causal:1d7adec21e9f279b7f58d571", + "stage": "verified", + "parentEventId": "causal:6daa1c387196cdd765473ab4", + "observer": "host", + "observedAt": 2, + "artifact": { + "deliveryHash": "fe7a64c4b079e5051ee322c3e14837fdd63480fbd3bfafa1150dae5d820e6a86", + "semanticDeltaHash": "38a6faf4120d6ff5418c07c74f17d7c1d4b6c5cb98b8a91d6fa0e5381de56e71", + "stage": "verified", + "trialId": "codex->claude-code:cross-model-001:0:runtime" + } + }, + { + "eventId": "causal:6cd986bc4290c089b35004f7", + "stage": "eligible", + "parentEventId": "causal:1d7adec21e9f279b7f58d571", + "observer": "host", + "observedAt": 3, + "artifact": { + "deliveryHash": "fe7a64c4b079e5051ee322c3e14837fdd63480fbd3bfafa1150dae5d820e6a86", + "semanticDeltaHash": "38a6faf4120d6ff5418c07c74f17d7c1d4b6c5cb98b8a91d6fa0e5381de56e71", + "stage": "eligible", + "trialId": "codex->claude-code:cross-model-001:0:runtime" + } + }, + { + "eventId": "causal:82f605cb8f84379d5d04ef47", + "stage": "retrieved", + "parentEventId": "causal:6cd986bc4290c089b35004f7", + "observer": "host", + "observedAt": 4, + "artifact": { + "deliveryHash": "fe7a64c4b079e5051ee322c3e14837fdd63480fbd3bfafa1150dae5d820e6a86", + "semanticDeltaHash": "38a6faf4120d6ff5418c07c74f17d7c1d4b6c5cb98b8a91d6fa0e5381de56e71", + "stage": "retrieved", + "trialId": "codex->claude-code:cross-model-001:0:runtime" + } + }, + { + "eventId": "causal:4edf1cbd6fd24de0da380ea9", + "stage": "delivered", + "parentEventId": "causal:82f605cb8f84379d5d04ef47", + "observer": "host", + "observedAt": 5, + "artifact": { + "deliveryHash": "fe7a64c4b079e5051ee322c3e14837fdd63480fbd3bfafa1150dae5d820e6a86", + "semanticDeltaHash": "38a6faf4120d6ff5418c07c74f17d7c1d4b6c5cb98b8a91d6fa0e5381de56e71", + "stage": "delivered", + "trialId": "codex->claude-code:cross-model-001:0:runtime" + } + }, + { + "eventId": "causal:0aa7c455904ee38a09667ad5", + "stage": "used", + "parentEventId": "causal:4edf1cbd6fd24de0da380ea9", + "observer": "host", + "observedAt": 6, + "artifact": { + "deliveryHash": "fe7a64c4b079e5051ee322c3e14837fdd63480fbd3bfafa1150dae5d820e6a86", + "semanticDeltaHash": "38a6faf4120d6ff5418c07c74f17d7c1d4b6c5cb98b8a91d6fa0e5381de56e71", + "stage": "used", + "trialId": "codex->claude-code:cross-model-001:0:runtime" + } + }, + { + "eventId": "causal:8558badde8a3cac43d2e6cd5", + "stage": "behaviorChanged", + "parentEventId": "causal:0aa7c455904ee38a09667ad5", + "observer": "host", + "observedAt": 7, + "artifact": { + "controlOutcomeHash": "d2822f3c62a3b22fa324dbd3d7b10f8d7c4d22514fd761c25bc6054f484a3f38", + "treatmentOutcomeHash": "1e97c501fe4fba8c189f999785d7516e2180a539ec3ea403d922b494ac75997e" + } + }, + { + "eventId": "causal:57c71807583938453dcb1146", + "stage": "mistakePrevented", + "parentEventId": "causal:8558badde8a3cac43d2e6cd5", + "observer": "host", + "observedAt": 8, + "artifact": { + "controlMistakeExecuted": true, + "treatmentMistakeExecuted": false + } + }, + { + "eventId": "causal:72ff837c93d3a3a74cdbf63b", + "stage": "taskCorrect", + "parentEventId": "causal:57c71807583938453dcb1146", + "observer": "host", + "observedAt": 9, + "artifact": { + "correct": true, + "graderReceiptHash": "f29f8f6c5039f7951211722dba9c863d6ed5a477b0c82b2f3b41ffe50b249ffc" + } + } + ], + "evidenceLedgerHash": "9fdfa19490960cf17d24d64dbd8c4335dad336fd0e2a9a50307b9fbfd3e6c373", + "controlOutcomeHash": "d2822f3c62a3b22fa324dbd3d7b10f8d7c4d22514fd761c25bc6054f484a3f38", + "treatmentOutcomeHash": "1e97c501fe4fba8c189f999785d7516e2180a539ec3ea403d922b494ac75997e", + "pairedPromptHash": "5c3b7ce785c01a18c505c885b6d255df00beb2ba6a520da87416cabbed4e53fb", + "graderReceiptHash": "f29f8f6c5039f7951211722dba9c863d6ed5a477b0c82b2f3b41ffe50b249ffc" + }, + "causalChainValid": true, + "outcomeHash": "1e97c501fe4fba8c189f999785d7516e2180a539ec3ea403d922b494ac75997e", + "workspaceStateHash": "229eac981143188f291b801bfe8fab3d642bf0a3b71f551798f53c2debc62e48", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "f29f8f6c5039f7951211722dba9c863d6ed5a477b0c82b2f3b41ffe50b249ffc", + "runnerExitCode": 0, + "runnerOutputHash": "8ec5f20a534b415439ab760ba62a0376f5f29d4f704076b513df0f6062244230", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-55446bee60219f2c", + "sequence": 1, + "previousHash": "48fcb9562a387c34ce971096043a0c7de5129d14bc051e55f194f5b129075bd8", + "rowHash": "203e34f2fa24a4bb7afff5489bb23bfd60c12e8a1979dacbd611bef3fbcf389b" + } + ], + "rowCount": 2, + "chainHead": "203e34f2fa24a4bb7afff5489bb23bfd60c12e8a1979dacbd611bef3fbcf389b", + "transcriptsPublished": false, + "endedAt": "2026-08-10T14:59:33.758Z", + "ledgerHash": "2470c21e2b53081f6afab645025864b50b015336d816c8c38517ee024ba1cdbb", + "signature": "+Xa4se77d6YVvEFq4DgSurhXDZpqqU+xR8FQjndLPLSJT/GAno1k6ZnSH2dBhyTGQwaw289U8zm4KdqW8bdVCg==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "2470c21e2b53081f6afab645025864b50b015336d816c8c38517ee024ba1cdbb", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "2470c21e2b53081f6afab645025864b50b015336d816c8c38517ee024ba1cdbb" + ], + "rejectedLedgers": 0, + "derivedHash": "a2aaea46ba70a41862db4c3097c48606e5519f91fc2b2b809e4101c0491beba7" + }, + "costDiagnostics": { + "passed": true, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [ + { + "dimension": "direction", + "value": "codex->claude-code", + "pairs": 1, + "tokenOverhead": -0.26170897390060777, + "latencyOverheadP95": 0.02020062453367231, + "regressions": [], + "passed": true + }, + { + "dimension": "family", + "value": "cross-model-handoff", + "pairs": 1, + "tokenOverhead": -0.26170897390060777, + "latencyOverheadP95": 0.02020062453367231, + "regressions": [], + "passed": true + }, + { + "dimension": "client", + "value": "claude-code", + "pairs": 1, + "tokenOverhead": -0.26170897390060777, + "latencyOverheadP95": 0.02020062453367231, + "regressions": [], + "passed": true + } + ] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "200ea919017a1e7a10eea3d40a09860f2f871c47a4e83030d250088769686fc7" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v4-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v4-attempts.jsonl new file mode 100644 index 000000000..734f9ef4e --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v4-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":[],"rowHash":"1a38be9580010456b6f5c206856e9f94f8c6634b9cf3b053c08445c5045764fe","runnerExitCode":0,"trialId":"codex->claude-code:cross-model-001:0:empty","valid":true,"validation":[]} +{"arm":"runtime","causalEvents":6,"driverValidation":[],"rowHash":"b84a53db639301b57b26608904a8b7f6464568c26ac202135dd254b5db03222f","runnerExitCode":0,"trialId":"codex->claude-code:cross-model-001:0:runtime","valid":true,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v4.json b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v4.json new file mode 100644 index 000000000..cd69a9715 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v4.json @@ -0,0 +1,639 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T15:04:31.638Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "gemini-cli 0.28.1", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialCount": 693 + }, + "coverage": { + "passed": true, + "checks": { + "effectivenessDesign": true, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [] + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "codex->claude-code", + "families": "cross-model-handoff", + "arms": "runtime,empty", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "6f7f5e0bd8a59e11f69978a7c36109a50dbaa42854bef10f313a6263be46e921" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": true, + "trialIntegrity": true, + "taskCorrect": true, + "runtimeDelivered": true, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": false + }, + "failed": [ + "tokenNonInferiority" + ], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": 0.30118847803604426 + }, + "failures": [], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-b41b295a817364f0", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "04b94602e8f05cf10b372097f7a937202ac4d56d4e3d2369cad98a926e000497", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T15:04:31.633Z", + "runHash": "661458c6fbe2ba24297adbe47b1ed49c48d7474d1e24674d0a2b3b34064b5b5c" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "codex->claude-code:cross-model-001:0:empty", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 37, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "empty", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "4f5777767e446baf7e658dd317134ebde2f555950387b8ac1a67e66459a5aeb8", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 2, + "totalTokens": 58983, + "latencyMs": 73607, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "c14fc55e509c4eb3b08f8912c304e110d0be7bc21e0f7a1e4b49ea15b1a703d7", + "workspaceStateHash": "56dcb01e34f92be97210737301db50575ba3c1f9c502398527220b54e5ccc2fa", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "c5d39c2daa8a0fc5bb37f839710a52d3730be76a280b74b2a7935fed1f6c02dc", + "runnerExitCode": 0, + "runnerOutputHash": "9d5fd02c3be8a0c6db4062d844ad32291b1e451582251d339c3ec85a005a6000", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-b41b295a817364f0", + "sequence": 0, + "previousHash": "661458c6fbe2ba24297adbe47b1ed49c48d7474d1e24674d0a2b3b34064b5b5c", + "rowHash": "60afdb50fe7763ed3bc3c74baffcd347ad7ab901bbe9f66f6cd10e8a99f15ae5" + }, + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "codex->claude-code:cross-model-001:0:runtime", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 340, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "runtime", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "90bc20152c4ca188e4703921bb2a20cb5dddf276e8b99f98ca953604448f538a", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": true, + "deliveryPhase": "pre-action", + "stale": false, + "contradictory": false, + "contextOverheadRatio": 186.5, + "reconstructionTokens": 2, + "totalTokens": 76748, + "latencyMs": 79619, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 373, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "7da230d987e97929604b0dcdc4958e766bfaaac90593aee35f23852fb612ceba", + "workspaceStateHash": "2657550a9eeabdc97f4b36b36270d739a29510c47d85bd2081c266cf99bfd460", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "d4efa61c267d3bcb53d660007965cc517eeb93090a0b9dbc551c371017534f92", + "runnerExitCode": 0, + "runnerOutputHash": "b07bd1514dd791d327aa3fc517abe5bb3986c98d51dea211d825a4121a8d35dd", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-b41b295a817364f0", + "sequence": 1, + "previousHash": "60afdb50fe7763ed3bc3c74baffcd347ad7ab901bbe9f66f6cd10e8a99f15ae5", + "rowHash": "a74f8db3a2de748ea4efb34ce3059eed03993dbeda42631726800226019793ca" + } + ], + "rowCount": 2, + "chainHead": "a74f8db3a2de748ea4efb34ce3059eed03993dbeda42631726800226019793ca", + "transcriptsPublished": false, + "endedAt": "2026-08-10T15:04:31.634Z", + "ledgerHash": "52159aeaf25ac2da45ad79ba150beb90f117a9d589e31f960f4379d8b7b37cdf", + "signature": "B2r0UDdc5etoQgnfPvrC+yU9cZvorm8tWlM4NZFpC7qRNkMXTePBuaELtVXzqqSzCGI5YFD7Y7W10vDIo/57Ag==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "52159aeaf25ac2da45ad79ba150beb90f117a9d589e31f960f4379d8b7b37cdf", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "52159aeaf25ac2da45ad79ba150beb90f117a9d589e31f960f4379d8b7b37cdf" + ], + "rejectedLedgers": 0, + "derivedHash": "a2aaea46ba70a41862db4c3097c48606e5519f91fc2b2b809e4101c0491beba7" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [ + { + "dimension": "direction", + "value": "codex->claude-code", + "pairs": 1, + "tokenOverhead": 0.30118847803604426, + "latencyOverheadP95": 0.08167701441438993, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + }, + { + "dimension": "family", + "value": "cross-model-handoff", + "pairs": 1, + "tokenOverhead": 0.30118847803604426, + "latencyOverheadP95": 0.08167701441438993, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + }, + { + "dimension": "client", + "value": "claude-code", + "pairs": 1, + "tokenOverhead": 0.30118847803604426, + "latencyOverheadP95": 0.08167701441438993, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + } + ] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "9310db9a6c22dd9e66f60f91a092db51afdc3b38fb0c0d18e1c43870fc8de208" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v5-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v5-attempts.jsonl new file mode 100644 index 000000000..64a29b964 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v5-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":[],"providerFailures":[],"rowHash":"9846dcc40ea73c138555618b3887a854699207836a625b321ff78350fc54edc5","runnerExitCode":0,"trialId":"codex->claude-code:cross-model-001:0:empty","valid":true,"validation":[]} +{"arm":"runtime","causalEvents":6,"driverValidation":[],"providerFailures":[],"rowHash":"2f06319b34766318d48feb1a34ee08a1a366e32d942c4efd9948a2aa1d75a9af","runnerExitCode":0,"trialId":"codex->claude-code:cross-model-001:0:runtime","valid":true,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v5.json b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v5.json new file mode 100644 index 000000000..e843c3e27 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v5.json @@ -0,0 +1,674 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T16:11:21.478Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "gemini-cli 0.28.1", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": false, + "crossSession": true, + "crossProject": false, + "concurrentAgents": false, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.2857142857142857, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 2, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "codex->claude-code", + "families": null, + "arms": "empty,runtime", + "requestedArms": "empty,runtime", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "6f7f5e0bd8a59e11f69978a7c36109a50dbaa42854bef10f313a6263be46e921" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "passed", + "passed": true, + "checks": { + "selectionComplete": true, + "trialIntegrity": true, + "taskCorrect": true, + "runtimeDelivered": true, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": true + }, + "failed": [], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": -0.0015796488572227799 + }, + "failures": [], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-0c5b2d20d3b6b6cd", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "b9eec6eae3d1a7f92fae68a8bec9ec3cccad7e4617895626c5e23101a250b0d4", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T16:11:21.473Z", + "runHash": "9f79b8b0fc56d79d2e2ec55e557c4da3ec66a5755a8545331c2a0f07dfc80482" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "codex->claude-code:cross-model-001:0:empty", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 37, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "empty", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "4f5777767e446baf7e658dd317134ebde2f555950387b8ac1a67e66459a5aeb8", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 52524, + "totalTokens": 60773, + "latencyMs": 65929, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "e617a7435b961152e7adbd8c09e60166ab019e14764c9dce5d35b30e463282be", + "workspaceStateHash": "00fd653709757ef187cd5eab3b3f8a49f78b0096e7e2fc6e10faaaf69e05c6f5", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "f7f970266ff351cdcbf0254fba83dab99bdc182ca8e5737c80474555551d42fd", + "runnerExitCode": 0, + "runnerOutputHash": "c264ccd37e456599c86778e5454f5b9f4c13ea67a70376f4857537d7804ca439", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-0c5b2d20d3b6b6cd", + "sequence": 0, + "previousHash": "9f79b8b0fc56d79d2e2ec55e557c4da3ec66a5755a8545331c2a0f07dfc80482", + "rowHash": "0a1f82f155eeb87e1e18dfa0668f792f4c3925740f2ca7b4271e033568ac87d3" + }, + { + "study": "full-effectiveness", + "planHash": "8d0debfb4abd51a0da45754b1d232c17395fd5348a7c47971d2f69018dadbd93", + "trialId": "codex->claude-code:cross-model-001:0:runtime", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 340, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "runtime", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "90bc20152c4ca188e4703921bb2a20cb5dddf276e8b99f98ca953604448f538a", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": true, + "deliveryPhase": "pre-action", + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0.004742612469038613, + "reconstructionTokens": 52081, + "totalTokens": 60677, + "latencyMs": 72450, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 247, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "a32bdae0b22ad4b7b9c67f21be0ce750f55483aace35e902391ada9af6505471", + "workspaceStateHash": "dceefaaeaf50578326b4ecddd776f60cac6d5d78c2f879b53677597e0a3691c8", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "61b8a77c1b728bba9c3e09316325458ed229075d68693dcec2dc384e90483888", + "runnerExitCode": 0, + "runnerOutputHash": "b163f6f431d83292405bdb5ebd952418e11493713aae07786c9e0ba1537a46fa", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-0c5b2d20d3b6b6cd", + "sequence": 1, + "previousHash": "0a1f82f155eeb87e1e18dfa0668f792f4c3925740f2ca7b4271e033568ac87d3", + "rowHash": "f32538fd3f17f8b64de2d2229ddfe7ca46f06d831aa31d52c18d161a2afa2b72" + } + ], + "rowCount": 2, + "chainHead": "f32538fd3f17f8b64de2d2229ddfe7ca46f06d831aa31d52c18d161a2afa2b72", + "transcriptsPublished": false, + "endedAt": "2026-08-10T16:11:21.474Z", + "ledgerHash": "96f3c664cf2cfded2dc97c6bfcdf0746ceb1fc69cd66f54f46bcde0a1dea2264", + "signature": "1qHygAGfW5h44CD6eXX0BHPbYcGDDAX2aTOzLAemmOPS8z6B4QVcitFfarWQAcdbId6cMQ8RvcoiiLrlX9EeAQ==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "96f3c664cf2cfded2dc97c6bfcdf0746ceb1fc69cd66f54f46bcde0a1dea2264", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "96f3c664cf2cfded2dc97c6bfcdf0746ceb1fc69cd66f54f46bcde0a1dea2264" + ], + "rejectedLedgers": 0, + "derivedHash": "707e69496fcd948884ddf1eb4257dee37c690a9754a637540e8b077f8c917a09" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [ + { + "dimension": "direction", + "value": "codex->claude-code", + "pairs": 1, + "tokenOverhead": -0.0015796488572227799, + "latencyOverheadP95": 0.09890943287475921, + "regressions": [ + "latency" + ], + "passed": false + }, + { + "dimension": "family", + "value": "cross-model-handoff", + "pairs": 1, + "tokenOverhead": -0.0015796488572227799, + "latencyOverheadP95": 0.09890943287475921, + "regressions": [ + "latency" + ], + "passed": false + }, + { + "dimension": "client", + "value": "claude-code", + "pairs": 1, + "tokenOverhead": -0.0015796488572227799, + "latencyOverheadP95": 0.09890943287475921, + "regressions": [ + "latency" + ], + "passed": false + } + ] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "1e4c42a2f812712e500456b58d535beb0c953f4dc616f1d50754faa1cc570934" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v6-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v6-attempts.jsonl new file mode 100644 index 000000000..c6154de22 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v6-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":[],"providerFailures":[],"rowHash":"a2095a6d7b9e32c3f9f9f7c153ed53ed1c4870c2a73165ad6090e71afe322c8e","runnerExitCode":0,"trialId":"codex->claude-code:cross-model-001:0:empty","valid":true,"validation":[]} +{"arm":"runtime","causalEvents":6,"driverValidation":[],"providerFailures":[],"rowHash":"26fefecfe21f8056a51d41e1ebd3e9e8a29449f7a2b00333863b10b6d2477b9d","runnerExitCode":0,"trialId":"codex->claude-code:cross-model-001:0:runtime","valid":true,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v6.json b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v6.json new file mode 100644 index 000000000..e6f1a9db7 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v6.json @@ -0,0 +1,682 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T17:49:24.492Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook", + "transport": "codex-app-server" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook", + "transport": "claude-code" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "1.1.11", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook", + "transport": "antigravity" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": false, + "crossSession": true, + "crossProject": false, + "concurrentAgents": false, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.2857142857142857, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 2, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "codex->claude-code", + "families": null, + "arms": "empty,runtime", + "requestedArms": "empty,runtime", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "6f7f5e0bd8a59e11f69978a7c36109a50dbaa42854bef10f313a6263be46e921" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": true, + "trialIntegrity": true, + "taskCorrect": true, + "runtimeDelivered": true, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": false + }, + "failed": [ + "tokenNonInferiority" + ], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": 0.05556858147713951 + }, + "failures": [], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-ba591d8dcb3f311b", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "e5f0f6751893c915c957266789ad14969833a975adc0c8d0bf0b1b1f99258fca", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T17:49:24.487Z", + "runHash": "1c708313020f0cdfe381dc2493b318ac851ac9e49e7418614a9ef89ebb68403e" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "codex->claude-code:cross-model-001:0:empty", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 37, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "empty", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "4f5777767e446baf7e658dd317134ebde2f555950387b8ac1a67e66459a5aeb8", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 50960, + "totalTokens": 17060, + "latencyMs": 45271, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "a352e4047b2529983960c0da1fe682769d603fa06f296da68fc35ededda6f83e", + "workspaceStateHash": "964ed491ff20d15d76ae03fd4fdd4b4ed5846bcfe1c323d71d587fd3eaaf2213", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "1ac3c22033cb88a6805210f2de6a34f5abffc783c89dac94c8ebd58256c57eef", + "runnerExitCode": 0, + "runnerOutputHash": "6ded9fce9dd97697e2de8361b90d92e738514ad57cb2a5f8cd7dad03124fdd9d", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-ba591d8dcb3f311b", + "sequence": 0, + "previousHash": "1c708313020f0cdfe381dc2493b318ac851ac9e49e7418614a9ef89ebb68403e", + "rowHash": "7786d9fb2f1d52a534349232f3a29b62d5d62d6cd29e765bf5fe75ad7c979af3" + }, + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "codex->claude-code:cross-model-001:0:runtime", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 340, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "runtime", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "90bc20152c4ca188e4703921bb2a20cb5dddf276e8b99f98ca953604448f538a", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": true, + "deliveryPhase": "pre-action", + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0.004594323127859827, + "reconstructionTokens": 53762, + "totalTokens": 18008, + "latencyMs": 68473, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 247, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "5ad3c1ad1dddb781926c6c508e9c6ed574b2938ed67cdb6e6e6129732d130966", + "workspaceStateHash": "f45e5b5fe41716e5d63a19ea3a0cf46d184c209cb29b6327d7b2ce8269e31c84", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "a8cfc53c69954d80bb2f253f50236a65c6c3d34cee453e721a240960a92bea8e", + "runnerExitCode": 0, + "runnerOutputHash": "29202f7d3aaafed4c7214f545635d2c08abcce2a34dbd86c8aa6b94fa87c1d13", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-ba591d8dcb3f311b", + "sequence": 1, + "previousHash": "7786d9fb2f1d52a534349232f3a29b62d5d62d6cd29e765bf5fe75ad7c979af3", + "rowHash": "8d4db8d48d2c0c0e0586d3024fa88e4480944b79347e274595cebd77834e7991" + } + ], + "rowCount": 2, + "chainHead": "8d4db8d48d2c0c0e0586d3024fa88e4480944b79347e274595cebd77834e7991", + "transcriptsPublished": false, + "endedAt": "2026-08-10T17:49:24.488Z", + "ledgerHash": "24bdac984dd6e8de13c9a6ab3410072b8731ee0dbb29a2c68a16224f52e92fe2", + "signature": "g9/xAznwB93x0F4zO01qRGOlWsWGqw+s60TKO4tY5w9w+KPGiZ98sq4ehbbINtdTiAr3gucXzNHGKQ32h2ceDA==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "24bdac984dd6e8de13c9a6ab3410072b8731ee0dbb29a2c68a16224f52e92fe2", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "24bdac984dd6e8de13c9a6ab3410072b8731ee0dbb29a2c68a16224f52e92fe2" + ], + "rejectedLedgers": 0, + "derivedHash": "101e5c1e72f61852ee2311e26a022b001dd6e708b74e2a54a631937a4e132d52" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [ + { + "dimension": "direction", + "value": "codex->claude-code", + "pairs": 1, + "tokenOverhead": 0.05556858147713951, + "latencyOverheadP95": 0.5125135296326566, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + }, + { + "dimension": "family", + "value": "cross-model-handoff", + "pairs": 1, + "tokenOverhead": 0.05556858147713951, + "latencyOverheadP95": 0.5125135296326566, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + }, + { + "dimension": "client", + "value": "claude-code", + "pairs": 1, + "tokenOverhead": 0.05556858147713951, + "latencyOverheadP95": 0.5125135296326566, + "regressions": [ + "tokens", + "latency" + ], + "passed": false + } + ] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "e00f920cb44a5e660642a576f807c7c9a1405eec65b65be05951ccec71dc6a92" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v7-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v7-attempts.jsonl new file mode 100644 index 000000000..8958c62ab --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v7-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":["unsupported study driver protocol","producer client mismatch","consumer client mismatch","trial requires exactly 2 planned provider calls","active model did not author semantic capture","model-authored semantic delta is incomplete","semantic delta hash mismatch","semantic delta is not bound to the producer invocation","semantic capture added a model call","semantic capture lacks verified repository evidence","consumer was exposed to the MCP server","consumer received static MCP schema tokens","host action audit is missing","host action audit is incomplete","provider-native total token accounting is missing","provider latency accounting is missing","executed CLI versions differ from the frozen plan","executed model version differs from the frozen plan","per-invocation phase telemetry is incomplete","execution topology producerContinuitySessionId mismatch","execution topology consumerContinuitySessionId mismatch","execution topology producerProjectId mismatch","execution topology consumerProjectId mismatch","successor agent topology mismatch","consumer invocations are not bound to planned agents"],"providerFailures":[],"rowHash":"1da5e94068297b0ba13f566ea7e8a19ac767fe9631df551585e296c583aaf8fa","runnerExitCode":null,"trialId":"codex->claude-code:cross-model-001:0:empty","valid":false,"validation":["result is not bound to its planned trial","trial integrity hash mismatch","hidden task variant mismatch","grader binding mismatch","workspace isolation mismatch","session isolation mismatch","executed promptHash differs from the planned trial","executed permissionsHash differs from the planned trial","executed budgets differs from the planned trial"]} +{"arm":"runtime","causalEvents":0,"driverValidation":["unsupported study driver protocol","producer client mismatch","consumer client mismatch","trial requires exactly 2 planned provider calls","active model did not author semantic capture","model-authored semantic delta is incomplete","semantic delta hash mismatch","semantic delta is not bound to the producer invocation","semantic capture added a model call","semantic capture lacks verified repository evidence","consumer was exposed to the MCP server","consumer received static MCP schema tokens","host action audit is missing","host action audit is incomplete","provider-native total token accounting is missing","provider latency accounting is missing","executed CLI versions differ from the frozen plan","executed model version differs from the frozen plan","per-invocation phase telemetry is incomplete","execution topology producerContinuitySessionId mismatch","execution topology consumerContinuitySessionId mismatch","execution topology producerProjectId mismatch","execution topology consumerProjectId mismatch","successor agent topology mismatch","consumer invocations are not bound to planned agents"],"providerFailures":[],"rowHash":"73a49c8e1cdf3e7eaa16013f56fe7f010c84eb88043b61f2df859262a557e51e","runnerExitCode":null,"trialId":"codex->claude-code:cross-model-001:0:runtime","valid":false,"validation":["result is not bound to its planned trial","trial integrity hash mismatch","hidden task variant mismatch","grader binding mismatch","workspace isolation mismatch","session isolation mismatch","executed promptHash differs from the planned trial","executed permissionsHash differs from the planned trial","executed budgets differs from the planned trial"]} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v7.json b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v7.json new file mode 100644 index 000000000..09b750dec --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v7.json @@ -0,0 +1,746 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T18:33:04.342Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook", + "transport": "codex-app-server" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook", + "transport": "claude-code" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "1.1.11", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook", + "transport": "antigravity" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": false, + "crossSession": true, + "crossProject": false, + "concurrentAgents": false, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.2857142857142857, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 2, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "codex->claude-code", + "families": null, + "arms": "empty,runtime", + "requestedArms": "empty,runtime", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "6f7f5e0bd8a59e11f69978a7c36109a50dbaa42854bef10f313a6263be46e921" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": false, + "trialIntegrity": false, + "taskCorrect": false, + "runtimeDelivered": false, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": false + }, + "failed": [ + "selectionComplete", + "trialIntegrity", + "taskCorrect", + "runtimeDelivered", + "tokenNonInferiority" + ], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": null + }, + "failures": [ + { + "trialId": "codex->claude-code:cross-model-001:0:empty", + "arm": "empty", + "runnerExitCode": null, + "valid": false, + "validation": [ + "result is not bound to its planned trial", + "trial integrity hash mismatch", + "hidden task variant mismatch", + "grader binding mismatch", + "workspace isolation mismatch", + "session isolation mismatch", + "executed promptHash differs from the planned trial", + "executed permissionsHash differs from the planned trial", + "executed budgets differs from the planned trial" + ], + "driverValidation": [ + "unsupported study driver protocol", + "producer client mismatch", + "consumer client mismatch", + "trial requires exactly 2 planned provider calls", + "active model did not author semantic capture", + "model-authored semantic delta is incomplete", + "semantic delta hash mismatch", + "semantic delta is not bound to the producer invocation", + "semantic capture added a model call", + "semantic capture lacks verified repository evidence", + "consumer was exposed to the MCP server", + "consumer received static MCP schema tokens", + "host action audit is missing", + "host action audit is incomplete", + "provider-native total token accounting is missing", + "provider latency accounting is missing", + "executed CLI versions differ from the frozen plan", + "executed model version differs from the frozen plan", + "per-invocation phase telemetry is incomplete", + "execution topology producerContinuitySessionId mismatch", + "execution topology consumerContinuitySessionId mismatch", + "execution topology producerProjectId mismatch", + "execution topology consumerProjectId mismatch", + "successor agent topology mismatch", + "consumer invocations are not bound to planned agents" + ], + "causalEvents": 0, + "providerFailures": [], + "rowHash": "1da5e94068297b0ba13f566ea7e8a19ac767fe9631df551585e296c583aaf8fa" + }, + { + "trialId": "codex->claude-code:cross-model-001:0:runtime", + "arm": "runtime", + "runnerExitCode": null, + "valid": false, + "validation": [ + "result is not bound to its planned trial", + "trial integrity hash mismatch", + "hidden task variant mismatch", + "grader binding mismatch", + "workspace isolation mismatch", + "session isolation mismatch", + "executed promptHash differs from the planned trial", + "executed permissionsHash differs from the planned trial", + "executed budgets differs from the planned trial" + ], + "driverValidation": [ + "unsupported study driver protocol", + "producer client mismatch", + "consumer client mismatch", + "trial requires exactly 2 planned provider calls", + "active model did not author semantic capture", + "model-authored semantic delta is incomplete", + "semantic delta hash mismatch", + "semantic delta is not bound to the producer invocation", + "semantic capture added a model call", + "semantic capture lacks verified repository evidence", + "consumer was exposed to the MCP server", + "consumer received static MCP schema tokens", + "host action audit is missing", + "host action audit is incomplete", + "provider-native total token accounting is missing", + "provider latency accounting is missing", + "executed CLI versions differ from the frozen plan", + "executed model version differs from the frozen plan", + "per-invocation phase telemetry is incomplete", + "execution topology producerContinuitySessionId mismatch", + "execution topology consumerContinuitySessionId mismatch", + "execution topology producerProjectId mismatch", + "execution topology consumerProjectId mismatch", + "successor agent topology mismatch", + "consumer invocations are not bound to planned agents" + ], + "causalEvents": 0, + "providerFailures": [], + "rowHash": "73a49c8e1cdf3e7eaa16013f56fe7f010c84eb88043b61f2df859262a557e51e" + } + ], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-be88b4b5d2078514", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "afabc04f3c39488063c546d3830006e8100118b928fb11ef7e033965cfb7dfbd", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T18:33:04.337Z", + "runHash": "400b2f51039c5ed6a23d1d11de6867b452562a2926bce301da7d3dc4c8a78b35" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "codex->claude-code:cross-model-001:0:empty", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 37, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "empty", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "producerModelVersion": "gpt-5.6-sol", + "consumerModelVersion": "claude-sonnet-5", + "producerTransport": "codex-app-server", + "consumerTransport": "claude-code", + "model": "claude-sonnet-5", + "modelVersion": null, + "modelAttestations": [], + "consumerMcpExposed": null, + "actionAuditHash": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "semanticEvidenceVerification": null, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "poweredStratum": false, + "correct": false, + "severeHarm": false, + "mistakeExecuted": true, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": null, + "reconstructionTokens": null, + "totalTokens": null, + "latencyMs": null, + "phaseAccounting": null, + "costLedger": null, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "workspaceStateHash": "6de8211f3e1e248d09046144509ae91eac5caf087858cd1047755151c58fc101", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "73afe2c8078440c5dd5db55d92c23c3008dced5242d08809c0c1b5903f11c2e8", + "runnerExitCode": null, + "runnerOutputHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "runnerErrorHash": "da90e572e18fc2c4fbd7744a31f64b3d0a3df53efd8ab85276ba16ad195ca251", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-be88b4b5d2078514", + "sequence": 0, + "previousHash": "400b2f51039c5ed6a23d1d11de6867b452562a2926bce301da7d3dc4c8a78b35", + "rowHash": "5ed6740ab3df411791b0dc2404552df5a67c3324c859ca2f782f3111da7c03ac" + }, + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "codex->claude-code:cross-model-001:0:runtime", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 340, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "runtime", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "producerModelVersion": "gpt-5.6-sol", + "consumerModelVersion": "claude-sonnet-5", + "producerTransport": "codex-app-server", + "consumerTransport": "claude-code", + "model": "claude-sonnet-5", + "modelVersion": null, + "modelAttestations": [], + "consumerMcpExposed": null, + "actionAuditHash": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "semanticEvidenceVerification": null, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "poweredStratum": false, + "correct": false, + "severeHarm": false, + "mistakeExecuted": true, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": null, + "reconstructionTokens": null, + "totalTokens": null, + "latencyMs": null, + "phaseAccounting": null, + "costLedger": null, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "workspaceStateHash": "6de8211f3e1e248d09046144509ae91eac5caf087858cd1047755151c58fc101", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "c6effd86c7143121f728e6f788f59985bd9e17bf67438f4d957ca0aa999af2e3", + "runnerExitCode": null, + "runnerOutputHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "runnerErrorHash": "da90e572e18fc2c4fbd7744a31f64b3d0a3df53efd8ab85276ba16ad195ca251", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-be88b4b5d2078514", + "sequence": 1, + "previousHash": "5ed6740ab3df411791b0dc2404552df5a67c3324c859ca2f782f3111da7c03ac", + "rowHash": "f6fb5953b085f6f59127d06b95a37acffa6962d3d0acfaa8473f77258bb506ac" + } + ], + "rowCount": 2, + "chainHead": "f6fb5953b085f6f59127d06b95a37acffa6962d3d0acfaa8473f77258bb506ac", + "transcriptsPublished": false, + "endedAt": "2026-08-10T18:33:04.338Z", + "ledgerHash": "1f9b55eb554cc625195a946492d88ab507564afde920847a1763aaef42af3030", + "signature": "yIieVdmskkR4/tcLkp7Vy7ErjLDhFmneYHGEeCmh20XQ8FO4f0qTJbIeVQltU0Kecq0Q3+2M0ml2+WBVNYz1Cw==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "1f9b55eb554cc625195a946492d88ab507564afde920847a1763aaef42af3030", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "1f9b55eb554cc625195a946492d88ab507564afde920847a1763aaef42af3030" + ], + "rejectedLedgers": 0, + "derivedHash": "5ca0b61c33badfa20541e649807e28049fbf13a4aa18ee9f42909769b4fe5eca" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "44950ca983e227bcc78c42cc3ccb7b381f513aa036e6eafd1ac71406ed19fbe3" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v8-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v8-attempts.jsonl new file mode 100644 index 000000000..4af705b50 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v8-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":["unsupported study driver protocol","producer client mismatch","consumer client mismatch","trial requires exactly 2 planned provider calls","active model did not author semantic capture","model-authored semantic delta is incomplete","semantic delta hash mismatch","semantic delta is not bound to the producer invocation","semantic capture added a model call","semantic capture lacks verified repository evidence","consumer was exposed to the MCP server","consumer received static MCP schema tokens","host action audit is missing","host action audit is incomplete","provider-native total token accounting is missing","provider latency accounting is missing","executed CLI versions differ from the frozen plan","executed model version differs from the frozen plan","per-invocation phase telemetry is incomplete","execution topology producerContinuitySessionId mismatch","execution topology consumerContinuitySessionId mismatch","execution topology producerProjectId mismatch","execution topology consumerProjectId mismatch","successor agent topology mismatch","consumer invocations are not bound to planned agents"],"providerFailures":[],"rowHash":"ac903d19c483a90413063abff3b6a04d3d24bf7fca79cb23a31d59b550b3e977","runnerExitCode":null,"trialId":"codex->claude-code:cross-model-001:0:empty","valid":false,"validation":["result is not bound to its planned trial","trial integrity hash mismatch","hidden task variant mismatch","grader binding mismatch","workspace isolation mismatch","session isolation mismatch","executed promptHash differs from the planned trial","executed permissionsHash differs from the planned trial","executed budgets differs from the planned trial"]} +{"arm":"runtime","causalEvents":6,"driverValidation":[],"providerFailures":[],"rowHash":"fa645f5a10ec240a64d20dfc9bc77b2ccfc5781fc7a186abdacf12e2e912e91a","runnerExitCode":0,"trialId":"codex->claude-code:cross-model-001:0:runtime","valid":true,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v8.json b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v8.json new file mode 100644 index 000000000..3f61b2ec0 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v8.json @@ -0,0 +1,966 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T18:46:06.689Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook", + "transport": "codex-app-server" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook", + "transport": "claude-code" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "1.1.11", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook", + "transport": "antigravity" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": false, + "crossSession": true, + "crossProject": false, + "concurrentAgents": false, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.2857142857142857, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 2, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "codex->claude-code", + "families": null, + "arms": "empty,runtime", + "requestedArms": "empty,runtime", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "6f7f5e0bd8a59e11f69978a7c36109a50dbaa42854bef10f313a6263be46e921" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": false, + "trialIntegrity": false, + "taskCorrect": true, + "runtimeDelivered": true, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": false + }, + "failed": [ + "selectionComplete", + "trialIntegrity", + "tokenNonInferiority" + ], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": null + }, + "failures": [ + { + "trialId": "codex->claude-code:cross-model-001:0:empty", + "arm": "empty", + "runnerExitCode": null, + "valid": false, + "validation": [ + "result is not bound to its planned trial", + "trial integrity hash mismatch", + "hidden task variant mismatch", + "grader binding mismatch", + "workspace isolation mismatch", + "session isolation mismatch", + "executed promptHash differs from the planned trial", + "executed permissionsHash differs from the planned trial", + "executed budgets differs from the planned trial" + ], + "driverValidation": [ + "unsupported study driver protocol", + "producer client mismatch", + "consumer client mismatch", + "trial requires exactly 2 planned provider calls", + "active model did not author semantic capture", + "model-authored semantic delta is incomplete", + "semantic delta hash mismatch", + "semantic delta is not bound to the producer invocation", + "semantic capture added a model call", + "semantic capture lacks verified repository evidence", + "consumer was exposed to the MCP server", + "consumer received static MCP schema tokens", + "host action audit is missing", + "host action audit is incomplete", + "provider-native total token accounting is missing", + "provider latency accounting is missing", + "executed CLI versions differ from the frozen plan", + "executed model version differs from the frozen plan", + "per-invocation phase telemetry is incomplete", + "execution topology producerContinuitySessionId mismatch", + "execution topology consumerContinuitySessionId mismatch", + "execution topology producerProjectId mismatch", + "execution topology consumerProjectId mismatch", + "successor agent topology mismatch", + "consumer invocations are not bound to planned agents" + ], + "causalEvents": 0, + "providerFailures": [], + "rowHash": "ac903d19c483a90413063abff3b6a04d3d24bf7fca79cb23a31d59b550b3e977" + } + ], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-b1b644bd89b647a9", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "137e9d805afcb43fba9d32d816ac547937374076a713c113961ced1bd2b3a627", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T18:46:06.684Z", + "runHash": "3682140878d825945b14378a68096e68dcb9b38a9f117bd6a58a972ed4fe7d6f" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "codex->claude-code:cross-model-001:0:empty", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 37, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "empty", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "producerModelVersion": "gpt-5.6-sol", + "consumerModelVersion": "claude-sonnet-5", + "producerTransport": "codex-app-server", + "consumerTransport": "claude-code", + "model": "claude-sonnet-5", + "modelVersion": null, + "modelAttestations": [], + "consumerMcpExposed": null, + "actionAuditHash": "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945", + "semanticEvidenceVerification": null, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "poweredStratum": false, + "correct": false, + "severeHarm": false, + "mistakeExecuted": true, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": null, + "reconstructionTokens": null, + "totalTokens": null, + "latencyMs": null, + "phaseAccounting": null, + "costLedger": null, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "workspaceStateHash": "6de8211f3e1e248d09046144509ae91eac5caf087858cd1047755151c58fc101", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "70f51d8b5c55d873e03d2e3345c9c5b55c386de73606c8140bec2e07322a955c", + "runnerExitCode": null, + "runnerOutputHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "runnerErrorHash": "da90e572e18fc2c4fbd7744a31f64b3d0a3df53efd8ab85276ba16ad195ca251", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-b1b644bd89b647a9", + "sequence": 0, + "previousHash": "3682140878d825945b14378a68096e68dcb9b38a9f117bd6a58a972ed4fe7d6f", + "rowHash": "fee006b38a48a0b1351bfe80e7ed0e8166954c8b3ded5abe5a87f1d7a8767955" + }, + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "codex->claude-code:cross-model-001:0:runtime", + "pairId": "codex->claude-code:cross-model-001:0", + "studySequence": 340, + "taskId": "cross-model-001", + "family": "cross-model-handoff", + "arm": "runtime", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "producerModelVersion": "gpt-5.6-sol", + "consumerModelVersion": "claude-sonnet-5", + "producerTransport": "codex-app-server", + "consumerTransport": "claude-code", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "modelAttestations": [ + { + "role": "producer", + "transport": "codex-app-server", + "providerRequestId": "019fecfe-219d-7f31-9682-a2b856a10d5d", + "attestation": { + "attestationHash": "976967b9d27bcaa980b21da2bc62c4ae13368db470459fc011910535447a432d", + "client": "codex", + "effectiveModel": "gpt-5.6-sol", + "evidenceHash": "50adb251f2af1f51f7adae0d130d138626ee417340bdd630db93fa19d4f6da67", + "provider": "openai", + "providerRequestId": "019fecfe-219d-7f31-9682-a2b856a10d5d", + "requestedModel": "gpt-5.6-sol", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "codex-app-server/thread-start" + } + }, + { + "role": "consumer-1", + "transport": "claude-code", + "providerRequestId": "199ec2d2-bfa5-4f86-b008-aa95384095b5", + "attestation": { + "attestationHash": "2593a665883a67fdaea5f99eef8273465b3eb2db7b1c5e786b62184e994993d4", + "client": "claude-code", + "effectiveModel": "claude-sonnet-5", + "evidenceHash": "014726eecad976ce2b223af35a64f11355333f613533065f153e5243743da503", + "provider": "anthropic", + "providerRequestId": "199ec2d2-bfa5-4f86-b008-aa95384095b5", + "requestedModel": "claude-sonnet-5", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "claude-code/stream-json" + } + } + ], + "consumerMcpExposed": false, + "actionAuditHash": "507f329402ef621390e78545d3ba856ae74f8c50d08da7832878737220d8ea83", + "semanticEvidenceVerification": { + "anchors": [ + { + "bytes": 123, + "path": ".ucr/handoff.json", + "sha256": "5c2d0201e3424f3388eab6a878bd6546da6ec7118551ac591c512c54a4e874b2" + }, + { + "bytes": 97, + "path": "evidence/current.json", + "sha256": "949f9f8238320be95cd4b1920122ed6b49ad9682590ed2e1ab659833737e04bd" + }, + { + "bytes": 210, + "path": "evidence/history.json", + "sha256": "8ce6b03ac4cbcd0c61754873eb08b5573045335a9dbe612f50f2e961b35ec739" + }, + { + "bytes": 87, + "path": "distractors/candidate-1.json", + "sha256": "a9e9d0dbdce8c5939ccfe918ce003610e8c528da41c1c967f33a57a699c3c747" + }, + { + "bytes": 87, + "path": "distractors/candidate-2.json", + "sha256": "3ed2f0cd27945642186c9c3a2114951fe609ba94e311ef2ff66885222083cebf" + }, + { + "bytes": 87, + "path": "distractors/candidate-3.json", + "sha256": "0038927ea87bf6c3e359bf5862bd9ae5ba6bd0026e7d13fcf16c537594253e3c" + }, + { + "bytes": 87, + "path": "distractors/candidate-4.json", + "sha256": "2a13c3cf5bec849f4cdbf572450b5c6f73ac14337d33b15ea552dbad8e9b85ed" + }, + { + "bytes": 87, + "path": "distractors/candidate-5.json", + "sha256": "16691ab4427051af678bb2384317cd62ddc302fdead9c081a212c8776f9abf31" + } + ], + "diagnostics": [ + "evidence path does not exist: TASK.mdtldr/summary", + "evidence path does not exist: TASK.mdtldr/analysis", + "evidence path does not exist: TASK.mdtldr/analysis2", + "evidence path does not exist: TASK.mdtldr/final" + ], + "schemaVersion": "ucr.semantic-evidence-verification/1", + "verificationHash": "1570ffed668001ca68f7529f962e6bdac2cbc978603aec3e1f13738bb8211ca0", + "verified": true + }, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "cross-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "90bc20152c4ca188e4703921bb2a20cb5dddf276e8b99f98ca953604448f538a", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": true, + "selected": true, + "delivered": true, + "deliveryPhase": "pre-action", + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0.002266737924552642, + "reconstructionTokens": 54263, + "totalTokens": 18809, + "latencyMs": 61591, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 123, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": true, + "causalChain": { + "stages": [ + { + "stage": "captured", + "eventId": "causal:4d668b4bad17e519cfe1ac29", + "parentEventId": null, + "artifactHash": "0f1094513ac5c37e13dda60762c4c1f6e55abd33c5d55f70ea888ade01b13dd3" + }, + { + "stage": "verified", + "eventId": "causal:1300a98e903ce07884062daf", + "parentEventId": "causal:4d668b4bad17e519cfe1ac29", + "artifactHash": "ccfe95476ffab1d5fd846a0a7f6afa7178687a8337ce361756afd5c1fb078f54" + }, + { + "stage": "eligible", + "eventId": "causal:92cc28c7c57d7811722ebb5e", + "parentEventId": "causal:1300a98e903ce07884062daf", + "artifactHash": "6ccf7de239a7daf2ae5b928b00d96b292bca02178597f711a3006b424f140a99" + }, + { + "stage": "retrieved", + "eventId": "causal:40ac427936588559b9f40ec8", + "parentEventId": "causal:92cc28c7c57d7811722ebb5e", + "artifactHash": "80ec5be67afdd40afc9ea944e083bbb5698f494f6753c604e02884ea0214d82a" + }, + { + "stage": "delivered", + "eventId": "causal:5410a73a8c7e469317abf3a0", + "parentEventId": "causal:40ac427936588559b9f40ec8", + "artifactHash": "4127910aa1002555a34d65a5ec7d61fefa634d84fbf1ecc1c24823326786fbb8" + }, + { + "stage": "used", + "eventId": "causal:80edd38a433f0291bc718b11", + "parentEventId": "causal:5410a73a8c7e469317abf3a0", + "artifactHash": "318b523fb7079fc1f016ac8ee8b09f832a33bbcdb9e1a5a9da738e53315332fd" + }, + { + "stage": "behaviorChanged", + "eventId": "causal:93d1f3e48fa6959bb6a72fc5", + "parentEventId": "causal:80edd38a433f0291bc718b11", + "artifactHash": "3e5b84848626ec2c44b941378beb6138b58f79133f2badcef163922b5df67de1" + }, + { + "stage": "mistakePrevented", + "eventId": "causal:57c71807583938453dcb1146", + "parentEventId": "causal:93d1f3e48fa6959bb6a72fc5", + "artifactHash": "6e2aa189dfd32ffee6fede58adeef7f5a25b9a8cbce27bdf0579a9c462b1b164" + }, + { + "stage": "taskCorrect", + "eventId": "causal:03896a3f754e403b2acd6260", + "parentEventId": "causal:57c71807583938453dcb1146", + "artifactHash": "e0be6bf6d2b989c808586e1c25a2434b5b58245226460deadea6c6f7effda9b6" + } + ], + "events": [ + { + "eventId": "causal:4d668b4bad17e519cfe1ac29", + "stage": "captured", + "parentEventId": null, + "observer": "host", + "observedAt": 1, + "artifact": { + "deliveryHash": "fe417259d224e724a8625353c6358befbf96f179cf703a15cd361b66967b56a3", + "semanticDeltaHash": "3fc92f2b8538c841733fd01a6f38927d672678e9c9f151fa690c1914a3ea7b5f", + "stage": "captured", + "trialId": "codex->claude-code:cross-model-001:0:runtime" + } + }, + { + "eventId": "causal:1300a98e903ce07884062daf", + "stage": "verified", + "parentEventId": "causal:4d668b4bad17e519cfe1ac29", + "observer": "host", + "observedAt": 2, + "artifact": { + "deliveryHash": "fe417259d224e724a8625353c6358befbf96f179cf703a15cd361b66967b56a3", + "semanticDeltaHash": "3fc92f2b8538c841733fd01a6f38927d672678e9c9f151fa690c1914a3ea7b5f", + "stage": "verified", + "trialId": "codex->claude-code:cross-model-001:0:runtime" + } + }, + { + "eventId": "causal:92cc28c7c57d7811722ebb5e", + "stage": "eligible", + "parentEventId": "causal:1300a98e903ce07884062daf", + "observer": "host", + "observedAt": 3, + "artifact": { + "deliveryHash": "fe417259d224e724a8625353c6358befbf96f179cf703a15cd361b66967b56a3", + "semanticDeltaHash": "3fc92f2b8538c841733fd01a6f38927d672678e9c9f151fa690c1914a3ea7b5f", + "stage": "eligible", + "trialId": "codex->claude-code:cross-model-001:0:runtime" + } + }, + { + "eventId": "causal:40ac427936588559b9f40ec8", + "stage": "retrieved", + "parentEventId": "causal:92cc28c7c57d7811722ebb5e", + "observer": "host", + "observedAt": 4, + "artifact": { + "deliveryHash": "fe417259d224e724a8625353c6358befbf96f179cf703a15cd361b66967b56a3", + "semanticDeltaHash": "3fc92f2b8538c841733fd01a6f38927d672678e9c9f151fa690c1914a3ea7b5f", + "stage": "retrieved", + "trialId": "codex->claude-code:cross-model-001:0:runtime" + } + }, + { + "eventId": "causal:5410a73a8c7e469317abf3a0", + "stage": "delivered", + "parentEventId": "causal:40ac427936588559b9f40ec8", + "observer": "host", + "observedAt": 5, + "artifact": { + "deliveryHash": "fe417259d224e724a8625353c6358befbf96f179cf703a15cd361b66967b56a3", + "semanticDeltaHash": "3fc92f2b8538c841733fd01a6f38927d672678e9c9f151fa690c1914a3ea7b5f", + "stage": "delivered", + "trialId": "codex->claude-code:cross-model-001:0:runtime" + } + }, + { + "eventId": "causal:80edd38a433f0291bc718b11", + "stage": "used", + "parentEventId": "causal:5410a73a8c7e469317abf3a0", + "observer": "host", + "observedAt": 6, + "artifact": { + "deliveryHash": "fe417259d224e724a8625353c6358befbf96f179cf703a15cd361b66967b56a3", + "semanticDeltaHash": "3fc92f2b8538c841733fd01a6f38927d672678e9c9f151fa690c1914a3ea7b5f", + "stage": "used", + "trialId": "codex->claude-code:cross-model-001:0:runtime" + } + }, + { + "eventId": "causal:93d1f3e48fa6959bb6a72fc5", + "stage": "behaviorChanged", + "parentEventId": "causal:80edd38a433f0291bc718b11", + "observer": "host", + "observedAt": 7, + "artifact": { + "controlOutcomeHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "treatmentOutcomeHash": "7b4269ad1c3a111aae869257cc076f3a2411a0d1f47f054f9677e4eb2984e095" + } + }, + { + "eventId": "causal:57c71807583938453dcb1146", + "stage": "mistakePrevented", + "parentEventId": "causal:93d1f3e48fa6959bb6a72fc5", + "observer": "host", + "observedAt": 8, + "artifact": { + "controlMistakeExecuted": true, + "treatmentMistakeExecuted": false + } + }, + { + "eventId": "causal:03896a3f754e403b2acd6260", + "stage": "taskCorrect", + "parentEventId": "causal:57c71807583938453dcb1146", + "observer": "host", + "observedAt": 9, + "artifact": { + "correct": true, + "graderReceiptHash": "09146c3a8d5437592c43d78b2f1ee5a6527d241253ee4c9805d7df14f2013da7" + } + } + ], + "evidenceLedgerHash": "3b43f78a179197b1b79c50ddd7aed1454017b43d5440b76e76312abc8170011a", + "controlOutcomeHash": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "treatmentOutcomeHash": "7b4269ad1c3a111aae869257cc076f3a2411a0d1f47f054f9677e4eb2984e095", + "pairedPromptHash": "5c3b7ce785c01a18c505c885b6d255df00beb2ba6a520da87416cabbed4e53fb", + "graderReceiptHash": "09146c3a8d5437592c43d78b2f1ee5a6527d241253ee4c9805d7df14f2013da7" + }, + "causalChainValid": true, + "outcomeHash": "7b4269ad1c3a111aae869257cc076f3a2411a0d1f47f054f9677e4eb2984e095", + "workspaceStateHash": "e6bf530fefd6dafcd456ce684fb687cde558932e63e566ba3ec3670d28246174", + "changedProtected": [], + "trialIntegrityValid": true, + "graderVerified": true, + "graderReceiptHash": "09146c3a8d5437592c43d78b2f1ee5a6527d241253ee4c9805d7df14f2013da7", + "runnerExitCode": 0, + "runnerOutputHash": "053c8deb995b58765d187a536f893829124346f24d5c0e257e725205122feb7f", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-b1b644bd89b647a9", + "sequence": 1, + "previousHash": "fee006b38a48a0b1351bfe80e7ed0e8166954c8b3ded5abe5a87f1d7a8767955", + "rowHash": "7d2919a8fc8466487dba14130fcdc7ad93a0446ab157d4b5f17a35d6211094e5" + } + ], + "rowCount": 2, + "chainHead": "7d2919a8fc8466487dba14130fcdc7ad93a0446ab157d4b5f17a35d6211094e5", + "transcriptsPublished": false, + "endedAt": "2026-08-10T18:46:06.684Z", + "ledgerHash": "587d40786cc33a9433046ddcb5409e11ebcd8ce9d6fe48f535fe7c95e6f0b163", + "signature": "HYp3oGVzbunanVthqLD1FKK/jksr6fqi6fAT8CKsQQpEZ/mQ0CvbLYpdsaSaLdIGxZ0too4qKAfd/G9il8SMAg==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "587d40786cc33a9433046ddcb5409e11ebcd8ce9d6fe48f535fe7c95e6f0b163", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "587d40786cc33a9433046ddcb5409e11ebcd8ce9d6fe48f535fe7c95e6f0b163" + ], + "rejectedLedgers": 0, + "derivedHash": "ce9b2d8bb08b607206d6880e057e4944205a5984364887e23e089a0faea8081a" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "9188537f44e42bef182aeaa4d94ab0f047a48aa058f4ab24d739b81d202f5796" +} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v9-attempts.jsonl b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v9-attempts.jsonl new file mode 100644 index 000000000..8b410aeda --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v9-attempts.jsonl @@ -0,0 +1,2 @@ +{"arm":"empty","causalEvents":0,"driverValidation":["semantic capture lacks verified repository evidence"],"providerFailures":[],"rowHash":"1690fdaf76fce13034fee87806fca1e26e37a3e5c4283d4b4929913f2da000b9","runnerExitCode":0,"trialId":"codex->claude-code:adversarial-001:0:empty","valid":false,"validation":[]} +{"arm":"runtime","causalEvents":0,"driverValidation":["semantic capture lacks verified repository evidence"],"providerFailures":[],"rowHash":"72daa2390dc0acdeb48dafa28b73a202223c5245b55b143a24732cb41ac8b538","runnerExitCode":0,"trialId":"codex->claude-code:adversarial-001:0:runtime","valid":false,"validation":[]} diff --git a/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v9.json b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v9.json new file mode 100644 index 000000000..30c29bc60 --- /dev/null +++ b/evals/ucr/results/superseded/full-study-qualification-codex-to-claude-v9.json @@ -0,0 +1,830 @@ +{ + "schemaVersion": "ucr.full-study/1", + "executedAt": "2026-08-10T18:51:49.254Z", + "evidenceClass": "executable-smoke", + "plan": { + "schemaVersion": "ucr.full-study-plan/1", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "seedHash": "41b25ff01f365e97e60404d48067231087bfdf3680c3243e04039cea20f40b62", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "modelVersion": "gpt-5.6-sol", + "lifecycleFamily": "process-hook", + "transport": "codex-app-server" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "modelVersion": "claude-sonnet-5", + "lifecycleFamily": "process-hook", + "transport": "claude-code" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelFamily": "google", + "version": "1.1.11", + "modelVersion": "gemini-2.5-pro", + "lifecycleFamily": "process-hook", + "transport": "antigravity" + } + ], + "directions": [ + "codex->codex", + "codex->claude-code", + "codex->gemini", + "claude-code->codex", + "claude-code->claude-code", + "claude-code->gemini", + "gemini->codex", + "gemini->claude-code", + "gemini->gemini" + ], + "pairedRepetitionsPerCell": 1, + "negativeRepetitionsPerCell": 1, + "includeSameClient": true, + "budgets": { + "contextTokens": 160, + "retries": 1, + "toolCalls": 40, + "timeoutMs": 600000 + }, + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "primaryPairsPerDirection": 11, + "preregisteredPairsPerArm": 357, + "minimumSubgroupPairs": 30, + "minimumNegativeSamplesPerDirection": 1053, + "negativeConfidence": { + "samples": 1053, + "maximumRate": 0.01, + "z": 3.2607674853887727, + "comparisons": 45, + "familyAlpha": 0.05, + "method": "bonferroni-familywise-wilson-zero-failure-upper-bound" + }, + "negativeSamplesPerDirectionPerArm": 11, + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialCount": 693 + }, + "coverage": { + "passed": false, + "checks": { + "effectivenessDesign": false, + "metricMapping": true, + "competitiveBaselines": true, + "productionStages": true + }, + "design": { + "passed": false, + "checks": { + "benchmarkFamilies": false, + "benchmarkArms": false, + "clients": false, + "modelFamilies": false, + "bidirectional": false, + "sameClient": false, + "crossClient": true, + "sameSession": true, + "crossSession": false, + "crossProject": false, + "concurrentAgents": false, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": false, + "executionTopology": true + }, + "coverage": { + "families": 0.09090909090909091, + "arms": 0.2857142857142857, + "familyCount": 1, + "requiredFamilies": 11, + "armCount": 2, + "requiredArms": 7, + "clients": 2, + "modelFamilies": 2, + "directions": 1, + "materialVariants": 1 + } + }, + "competitionCoverage": 1, + "productionCoverage": 1, + "metricSources": { + "applicabilityPrecision": "full-effectiveness", + "applicabilityPrecisionIntervalLow": "full-effectiveness", + "preActionDelivery": "full-effectiveness", + "preActionDeliveryIntervalLow": "full-effectiveness", + "irrelevantDelivery": "full-effectiveness", + "irrelevantDeliveryIntervalHigh": "full-effectiveness", + "staleDelivery": "full-effectiveness", + "staleDeliveryIntervalHigh": "full-effectiveness", + "recurrenceReduction": "full-effectiveness", + "recurrenceIntervalLow": "full-effectiveness", + "naturalCorrectnessDelta": "full-effectiveness", + "naturalCorrectnessIntervalLow": "full-effectiveness", + "severeUnquarantined": "full-effectiveness", + "emptyP95Overhead": "full-effectiveness", + "reconstructionTokenReduction": "full-effectiveness", + "firstSuccessorTokenReduction": "full-effectiveness", + "firstSuccessorTokenIntervalLow": "full-effectiveness", + "latencyOverheadP95": "full-effectiveness", + "knownMistakeRecurrence": "full-effectiveness", + "contradictoryDelivery": "full-effectiveness", + "contradictoryDeliveryIntervalHigh": "full-effectiveness", + "negativeDeliveryIntervalHigh": "full-effectiveness", + "consumerSchemaTokensP95": "full-effectiveness", + "captureModelCallsP95": "full-effectiveness", + "writerIntegrity": "writer-conformance", + "crossClientPassed": "full-effectiveness", + "benchmarkFamilyCoverage": "full-effectiveness", + "benchmarkArmCoverage": "full-effectiveness", + "directionalCorrectnessIntervalLow": "full-effectiveness", + "familyCorrectnessIntervalLow": "full-effectiveness", + "directionalTokenOverheadHigh": "full-effectiveness", + "causalChainIntegrity": "full-effectiveness", + "trialIntegrity": "full-effectiveness", + "independentGrading": "full-effectiveness", + "competitivePassed": "live-competitive", + "competitiveCoverage": "live-competitive", + "competitiveReproducibility": "live-competitive" + }, + "missingMetrics": [], + "basis": "selected-executed-trials", + "preflightDesign": { + "passed": true, + "checks": { + "benchmarkFamilies": true, + "benchmarkArms": true, + "clients": true, + "modelFamilies": true, + "bidirectional": true, + "sameClient": true, + "crossClient": true, + "sameSession": true, + "crossSession": true, + "crossProject": true, + "concurrentAgents": true, + "pairedControls": true, + "isolatedWorkspaces": true, + "materialVariants": true, + "executionTopology": true + }, + "coverage": { + "families": 1, + "arms": 1, + "familyCount": 11, + "requiredFamilies": 11, + "armCount": 7, + "requiredArms": 7, + "clients": 3, + "modelFamilies": 3, + "directions": 9, + "materialVariants": 485 + } + } + }, + "poweredDesign": false, + "complete": false, + "trialsPlanned": 693, + "trialsExecuted": 2, + "selection": { + "directions": "codex->claude-code", + "families": "adversarial-memory", + "arms": "empty,runtime", + "requestedArms": "empty,runtime", + "maxTrials": 2, + "selectedTrials": 2, + "selectedTrialIdsHash": "9dd613643546c2fe6770bd4a2cdb990a0a86750edb64ce949c2a4022919d788b" + }, + "qualification": { + "schemaVersion": "ucr.study-qualification/1", + "status": "failed", + "passed": false, + "checks": { + "selectionComplete": false, + "trialIntegrity": false, + "taskCorrect": true, + "runtimeDelivered": false, + "controlsWithheld": true, + "hardNegativesWithheld": true, + "pairedCoverage": true, + "correctnessNonInferiority": true, + "tokenNonInferiority": true + }, + "failed": [ + "selectionComplete", + "trialIntegrity", + "runtimeDelivered" + ], + "selectedTrials": 2, + "executedTrials": 2, + "completePairs": 1, + "maximumObservedTokenOverhead": 0.007990387503754881 + }, + "failures": [ + { + "trialId": "codex->claude-code:adversarial-001:0:empty", + "arm": "empty", + "runnerExitCode": 0, + "valid": false, + "validation": [], + "driverValidation": [ + "semantic capture lacks verified repository evidence" + ], + "causalEvents": 0, + "providerFailures": [], + "rowHash": "1690fdaf76fce13034fee87806fca1e26e37a3e5c4283d4b4929913f2da000b9" + }, + { + "trialId": "codex->claude-code:adversarial-001:0:runtime", + "arm": "runtime", + "runnerExitCode": 0, + "valid": false, + "validation": [], + "driverValidation": [ + "semantic capture lacks verified repository evidence" + ], + "causalEvents": 0, + "providerFailures": [], + "rowHash": "72daa2390dc0acdeb48dafa28b73a202223c5245b55b143a24732cb41ac8b538" + } + ], + "ledger": { + "schemaVersion": "ucr.evidence-ledger/2", + "evidenceClass": "executable-smoke", + "run": { + "schemaVersion": "ucr.evidence-run/2", + "runId": "full-study-d0a62dfe42b255a2", + "evidenceClass": "executable-smoke", + "benchmarkHash": "597b75c9fb2a20a171bdde67c7a40cc5aef9aa27ee730550a747c750a39243d2", + "sourceTreeHash": "8cc8fac69791cbeb3a7ef2d84c0b7ec5af02309f1dc68c14891c9c8a6748e5ad", + "runner": { + "name": "ucr-full-study", + "node": "v25.6.0", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "registrationHash": "e171d5da8431aa64f847b8bd5452b7f40facdb4330d0c21d6b6e38d89b486549", + "poweredDesign": false, + "minimumSubgroupPairs": 30 + }, + "startedAt": "2026-08-10T18:51:49.250Z", + "runHash": "5614ce57094d0fe8816e202fb6aaf3a277e3238617f00d6dcef8d340b99b4bec" + }, + "rows": [ + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "codex->claude-code:adversarial-001:0:empty", + "pairId": "codex->claude-code:adversarial-001:0", + "studySequence": 157, + "taskId": "adversarial-001", + "family": "adversarial-memory", + "arm": "empty", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "producerModelVersion": "gpt-5.6-sol", + "consumerModelVersion": "claude-sonnet-5", + "producerTransport": "codex-app-server", + "consumerTransport": "claude-code", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "modelAttestations": [ + { + "role": "producer", + "transport": "codex-app-server", + "providerRequestId": "019fed02-e381-7f33-9601-edbd01ecdcbd", + "attestation": { + "attestationHash": "bbaa2418df113a5990cef9f7e41f9484b866728f40f8118f7d80345440dca516", + "client": "codex", + "effectiveModel": "gpt-5.6-sol", + "evidenceHash": "43b19855b7831183afdb5341f61c70ff362f89ff3f30cf78682e6447eaa73b38", + "provider": "openai", + "providerRequestId": "019fed02-e381-7f33-9601-edbd01ecdcbd", + "requestedModel": "gpt-5.6-sol", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "codex-app-server/thread-start" + } + }, + { + "role": "consumer-1", + "transport": "claude-code", + "providerRequestId": "67e7eca1-4d7e-4839-8603-e9bf661a0991", + "attestation": { + "attestationHash": "2c0284666924e48531145e46eaf4d6cbec5011b31d20eab2ed377d97cbb49cab", + "client": "claude-code", + "effectiveModel": "claude-sonnet-5", + "evidenceHash": "e25f2a7b116770e0dabbb68f709fb98a92ea5abe9b0e2c950ea634e91cb83667", + "provider": "anthropic", + "providerRequestId": "67e7eca1-4d7e-4839-8603-e9bf661a0991", + "requestedModel": "claude-sonnet-5", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "claude-code/stream-json" + } + } + ], + "consumerMcpExposed": false, + "actionAuditHash": "a6eda7fcba2b970694a82a2265f2d9018a791aed0bf52f5b031990463f6732d3", + "semanticEvidenceVerification": { + "anchors": [ + { + "bytes": 97, + "path": "docs/verified-current.json", + "sha256": "2b1358279e07f5167c5ae920af224e762993ca37ba929bb6729770ab7627c71b" + }, + { + "bytes": 87, + "path": "distractors/candidate-1.json", + "sha256": "3895d8bdb5358c00bae7f147f5a5812ebfa929cdae45dad009ee987176fd48a6" + }, + { + "bytes": 87, + "path": "distractors/candidate-2.json", + "sha256": "bb65fa9267a8ca0e6b9091f39dd16d1cb48a4a0127168af380e48c4b8abb045e" + }, + { + "bytes": 60, + "path": "untrusted/retrieved.txt", + "sha256": "3f26bbdbe5428b4b85b54549afd0d9e39070567c2b4537337b14a501905ed615" + }, + { + "bytes": 210, + "path": "docs/decision-history.json", + "sha256": "2748b48002a6ea8ac247bed5864134cb9f00683cb30471e163165f4003e991bd" + } + ], + "diagnostics": [ + "evidence path does not exist: TASK.mddocs/decision-history.json", + "declared evidence ref was not verified: TASK.mddocs/decision-history.json" + ], + "schemaVersion": "ucr.semantic-evidence-verification/1", + "verificationHash": "1c10baf6b429eda7e4786ae13b489f3f991d6f478dc05c06bf838aaf219f7f42", + "verified": false + }, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "same-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "b0b618071fab05378bf7814d40979fdfc6899d1c66d9ea4b747270945fe2ed3a", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": false, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 49615, + "totalTokens": 16645, + "latencyMs": 48260, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "4128033fb542358ae2fb6fe63831dc24addf1c9ce3704b2d08c77fc30503f00b", + "workspaceStateHash": "6fe68b8bc7e3c4d7232b706ab905f70b39e8899228ddfd78df8a296c4979754d", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "800415ada10560bf3aba35f7f5beab5231d6c90b4bb065ca8a256555d069143e", + "runnerExitCode": 0, + "runnerOutputHash": "5e743b95da2a286e1ef62ddd5b05f73d067903c73570db044108904fdb441651", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-d0a62dfe42b255a2", + "sequence": 0, + "previousHash": "5614ce57094d0fe8816e202fb6aaf3a277e3238617f00d6dcef8d340b99b4bec", + "rowHash": "9a9bf2c7111a493c5e80dce638f46862593582b1884a30dc2036fb611171229a" + }, + { + "study": "full-effectiveness", + "planHash": "ea8fa07120d0046c15525d3311a47b0326a7309ce6c7403d3726a65c0791096f", + "trialId": "codex->claude-code:adversarial-001:0:runtime", + "pairId": "codex->claude-code:adversarial-001:0", + "studySequence": 259, + "taskId": "adversarial-001", + "family": "adversarial-memory", + "arm": "runtime", + "direction": "codex->claude-code", + "producerClient": "codex", + "consumerClient": "claude-code", + "producerFamily": "openai", + "consumerFamily": "anthropic", + "producerModel": "gpt-5.6-sol", + "consumerModel": "claude-sonnet-5", + "producerModelVersion": "gpt-5.6-sol", + "consumerModelVersion": "claude-sonnet-5", + "producerTransport": "codex-app-server", + "consumerTransport": "claude-code", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "modelAttestations": [ + { + "role": "producer", + "transport": "codex-app-server", + "providerRequestId": "019fed03-a1be-7b52-957a-042a8a5a5402", + "attestation": { + "attestationHash": "ca93fe3b29e84e3da4159134c80b98c1e030cddfb58b38195c9929c89f026de2", + "client": "codex", + "effectiveModel": "gpt-5.6-sol", + "evidenceHash": "cbcbb2d2e391590719db9d35cf04709053e5ccae1675efe50ed0b9eb2882ffa8", + "provider": "openai", + "providerRequestId": "019fed03-a1be-7b52-957a-042a8a5a5402", + "requestedModel": "gpt-5.6-sol", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "codex-app-server/thread-start" + } + }, + { + "role": "consumer-1", + "transport": "claude-code", + "providerRequestId": "6cf060c9-bc70-4602-8057-a0905ec8109d", + "attestation": { + "attestationHash": "f56aac1c6b4793bbccb87951a2801dde451e69ac80f98c7b6a237d7f15c34917", + "client": "claude-code", + "effectiveModel": "claude-sonnet-5", + "evidenceHash": "a91659c0159488b9e6d8ad685c426216612404231906eeab8c7c2e88a2281760", + "provider": "anthropic", + "providerRequestId": "6cf060c9-bc70-4602-8057-a0905ec8109d", + "requestedModel": "claude-sonnet-5", + "reroutes": [], + "schemaVersion": "ucr.model-attestation/1", + "source": "claude-code/stream-json" + } + } + ], + "consumerMcpExposed": false, + "actionAuditHash": "a1f7329510d29609141123f1b96acd4bcf5fe4705615a22251a881b2a4b8d639", + "semanticEvidenceVerification": { + "anchors": [ + { + "bytes": 97, + "path": "docs/verified-current.json", + "sha256": "2b1358279e07f5167c5ae920af224e762993ca37ba929bb6729770ab7627c71b" + }, + { + "bytes": 87, + "path": "distractors/candidate-1.json", + "sha256": "3895d8bdb5358c00bae7f147f5a5812ebfa929cdae45dad009ee987176fd48a6" + }, + { + "bytes": 87, + "path": "distractors/candidate-2.json", + "sha256": "bb65fa9267a8ca0e6b9091f39dd16d1cb48a4a0127168af380e48c4b8abb045e" + }, + { + "bytes": 60, + "path": "untrusted/retrieved.txt", + "sha256": "3f26bbdbe5428b4b85b54549afd0d9e39070567c2b4537337b14a501905ed615" + }, + { + "bytes": 210, + "path": "docs/decision-history.json", + "sha256": "2748b48002a6ea8ac247bed5864134cb9f00683cb30471e163165f4003e991bd" + } + ], + "diagnostics": [ + "evidence path does not exist: TASK.mddocs/decision-history.json", + "declared evidence ref was not verified: TASK.mddocs/decision-history.json" + ], + "schemaVersion": "ucr.semantic-evidence-verification/1", + "verificationHash": "1c10baf6b429eda7e4786ae13b489f3f991d6f478dc05c06bf838aaf219f7f42", + "verified": false + }, + "permissionsHash": "99cfc916f214c8e56d0093fa298b4e7f8739bd206e7452bbdfe25a9396338a38", + "contextBudget": 160, + "retryBudget": 1, + "toolBudget": 40, + "sessionMode": "same-session", + "projectMode": "same-project", + "agentMode": "single-successor", + "expectedProviderInvocations": 2, + "executionTopologyHash": "dcb288782d4a701b2eb3c12d4119ef2d9ef77596b1899d1b02baa444e7d4ea0c", + "poweredStratum": false, + "correct": true, + "severeHarm": false, + "mistakeExecuted": false, + "knownMistake": false, + "quarantinedBeforeNext": false, + "applicable": true, + "eligible": false, + "selected": false, + "delivered": false, + "deliveryPhase": null, + "stale": false, + "contradictory": false, + "contextOverheadRatio": 0, + "reconstructionTokens": 49676, + "totalTokens": 16778, + "latencyMs": 43577, + "phaseAccounting": { + "captureModelCalls": 0, + "contextTokens": 0, + "staticSchemaTokens": 0 + }, + "costLedger": { + "costUsd": null, + "source": "provider-cli-native" + }, + "providerFailures": [], + "causalClaim": false, + "causalChain": null, + "causalChainValid": false, + "outcomeHash": "b06e78b526590e855783a54c3be77b89106fd5cc32e78a0a133ad343ec313c1e", + "workspaceStateHash": "1a516c8e33e0ad538953fe410c3b6afcfdb374901b5c5b63a5a64242639884cc", + "changedProtected": [], + "trialIntegrityValid": false, + "graderVerified": true, + "graderReceiptHash": "f781eeef393b77a5d3f90092f9bf48fbaecb860977bb4d78def35214fd0822f1", + "runnerExitCode": 0, + "runnerOutputHash": "14df884f11f3e4e359964ee6755535cf5ba051e6aa54badeeb25702234984102", + "runnerErrorHash": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "schemaVersion": "ucr.evidence-row/2", + "runId": "full-study-d0a62dfe42b255a2", + "sequence": 1, + "previousHash": "9a9bf2c7111a493c5e80dce638f46862593582b1884a30dc2036fb611171229a", + "rowHash": "6ca68464f1d8a3c81436f11d6ff0dd1c78eb12e6feb6beff8df043b23ef16994" + } + ], + "rowCount": 2, + "chainHead": "6ca68464f1d8a3c81436f11d6ff0dd1c78eb12e6feb6beff8df043b23ef16994", + "transcriptsPublished": false, + "endedAt": "2026-08-10T18:51:49.250Z", + "ledgerHash": "a43793fa0bbc5c52afc12fb34662b9999b70fc6781b67773e8023d87a3afe6ae", + "signature": "m0wx0HNxLg73JjRBDA5rEYhyolnsFDGfA8bXi+o4ZMqzSiG1qdQ+cfQ5kxmXt6pctGFDsCNM/1GobrxErL81DA==" + }, + "ledgerKeyId": "ucr-ed25519-77b546af455c7ecd3c9c", + "ledgerVerification": { + "valid": true, + "diagnostics": [], + "validSignature": true, + "ledgerHash": "a43793fa0bbc5c52afc12fb34662b9999b70fc6781b67773e8023d87a3afe6ae", + "rowCount": 2 + }, + "derived": { + "schemaVersion": "ucr.derived-metrics/2", + "metrics": { + "applicabilityPrecision": null, + "applicabilityPrecisionIntervalLow": null, + "preActionDelivery": null, + "preActionDeliveryIntervalLow": null, + "irrelevantDelivery": null, + "irrelevantDeliveryIntervalHigh": null, + "staleDelivery": null, + "staleDeliveryIntervalHigh": null, + "recurrenceReduction": null, + "recurrenceIntervalLow": null, + "naturalCorrectnessDelta": null, + "naturalCorrectnessIntervalLow": null, + "severeUnquarantined": 0, + "emptyP95Overhead": null, + "reconstructionTokenReduction": null, + "firstSuccessorTokenReduction": null, + "firstSuccessorTokenIntervalLow": null, + "latencyOverheadP95": null, + "knownMistakeRecurrence": null, + "contradictoryDelivery": null, + "contradictoryDeliveryIntervalHigh": null, + "negativeDeliveryIntervalHigh": null, + "consumerSchemaTokensP95": null, + "captureModelCallsP95": null, + "writerIntegrity": false, + "crossClientPassed": false, + "benchmarkFamilyCoverage": null, + "benchmarkArmCoverage": null, + "directionalCorrectnessIntervalLow": null, + "familyCorrectnessIntervalLow": null, + "directionalTokenOverheadHigh": null, + "causalChainIntegrity": false, + "trialIntegrity": false, + "independentGrading": false, + "competitivePassed": false, + "competitiveCoverage": null, + "competitiveReproducibility": false + }, + "intervals": { + "recurrence": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "correctness": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "reconstruction": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + }, + "firstSuccessorTokens": { + "pairs": 0, + "mean": null, + "low": null, + "high": null + } + }, + "sources": { + "applicabilityPrecision": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "applicabilityPrecisionIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "preActionDeliveryIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "irrelevantDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "staleDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "recurrenceIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessDelta": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "naturalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "severeUnquarantined": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "emptyP95Overhead": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "reconstructionTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenReduction": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "firstSuccessorTokenIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "latencyOverheadP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "knownMistakeRecurrence": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDelivery": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "contradictoryDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "negativeDeliveryIntervalHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "consumerSchemaTokensP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "captureModelCallsP95": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "writerIntegrity": { + "requiredClass": "conformance", + "eligibleLedgerHashes": [] + }, + "crossClientPassed": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkFamilyCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "benchmarkArmCoverage": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "familyCorrectnessIntervalLow": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "directionalTokenOverheadHigh": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "causalChainIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "trialIntegrity": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "independentGrading": { + "requiredClass": "effectiveness", + "eligibleLedgerHashes": [] + }, + "competitivePassed": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveCoverage": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + }, + "competitiveReproducibility": { + "requiredClass": "superiority", + "eligibleLedgerHashes": [] + } + }, + "inputLedgerHashes": [ + "a43793fa0bbc5c52afc12fb34662b9999b70fc6781b67773e8023d87a3afe6ae" + ], + "rejectedLedgers": 0, + "derivedHash": "76d19c72906c681ef20898362cde7785acfaf686a4fc4e56dced346195647827" + }, + "costDiagnostics": { + "passed": false, + "thresholds": { + "maximumTokenOverhead": 0.05, + "maximumLatencyOverhead": 0.05 + }, + "groups": [] + }, + "passed": false, + "limitations": [ + "qualification or partial execution cannot supply effectiveness metrics" + ], + "reportHash": "b304d043f62cb2bfbf503d5901a435ca5d8b36f7cac706e1e4bbf7045d182325" +} diff --git a/evals/ucr/study-clients-gemini-api-v1.json b/evals/ucr/study-clients-gemini-api-v1.json new file mode 100644 index 000000000..14ec45ac5 --- /dev/null +++ b/evals/ucr/study-clients-gemini-api-v1.json @@ -0,0 +1,35 @@ +{ + "schemaVersion": "ucr.study-clients/1", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "transport": "codex-app-server", + "lifecycleFamily": "process-hook", + "commandEnvironment": "UCR_CODEX_COMMAND" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "transport": "claude-code", + "lifecycleFamily": "process-hook", + "commandEnvironment": "UCR_CLAUDE_COMMAND" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelVersion": "gemini-2.5-pro", + "modelFamily": "google", + "version": "gemini-cli 0.28.1", + "transport": "gemini-cli", + "lifecycleFamily": "process-hook", + "commandEnvironment": "UCR_GEMINI_COMMAND" + } + ] +} diff --git a/evals/ucr/study-clients-v1.json b/evals/ucr/study-clients-v1.json new file mode 100644 index 000000000..32c033f02 --- /dev/null +++ b/evals/ucr/study-clients-v1.json @@ -0,0 +1,35 @@ +{ + "schemaVersion": "ucr.study-clients/1", + "clients": [ + { + "id": "codex", + "model": "gpt-5.6-sol", + "modelVersion": "gpt-5.6-sol", + "modelFamily": "openai", + "version": "codex-cli 0.147.0", + "transport": "codex-app-server", + "lifecycleFamily": "process-hook", + "commandEnvironment": "UCR_CODEX_COMMAND" + }, + { + "id": "claude-code", + "model": "claude-sonnet-5", + "modelVersion": "claude-sonnet-5", + "modelFamily": "anthropic", + "version": "2.1.225 (Claude Code)", + "transport": "claude-code", + "lifecycleFamily": "process-hook", + "commandEnvironment": "UCR_CLAUDE_COMMAND" + }, + { + "id": "gemini", + "model": "gemini-2.5-pro", + "modelVersion": "gemini-2.5-pro", + "modelFamily": "google", + "version": "1.1.11", + "transport": "antigravity", + "lifecycleFamily": "process-hook", + "commandEnvironment": "UCR_GEMINI_COMMAND" + } + ] +} diff --git a/evals/ucr/trusted-evidence-keys/ucr-ed25519-77b546af455c7ecd3c9c.pem b/evals/ucr/trusted-evidence-keys/ucr-ed25519-77b546af455c7ecd3c9c.pem new file mode 100644 index 000000000..be4ef373e --- /dev/null +++ b/evals/ucr/trusted-evidence-keys/ucr-ed25519-77b546af455c7ecd3c9c.pem @@ -0,0 +1,3 @@ +-----BEGIN PUBLIC KEY----- +MCowBQYDK2VwAyEAQdZKDxd8FTd4hw1igDlYl/NbhaD/NxWILx+zex98NmU= +-----END PUBLIC KEY----- diff --git a/package.json b/package.json index 010c58a2e..d713bcc7b 100644 --- a/package.json +++ b/package.json @@ -2,7 +2,7 @@ "name": "@ooples/token-optimizer-mcp", "version": "5.6.1", "mcpName": "io.github.ooples/token-optimizer-mcp", - "description": "Intelligent context window optimization for Claude Code - store content externally via caching and compression, freeing up your context window for what matters", + "description": "Cross-client context optimization and durable live knowledge capture for AI coding agents and CLI tools", "main": "dist/server/index.js", "types": "dist/server/index.d.ts", "bin": { @@ -64,6 +64,10 @@ "verify:clients": "node scripts/verify-client-configs.mjs", "verify:certification": "node scripts/certify-clients.mjs --no-version-check", "verify:ucr": "node scripts/verify-ucr-runtime.mjs", + "verify:ucr:study-design": "node scripts/verify-ucr-study-readiness.mjs", + "eval:ucr:qualification:verify": "node scripts/verify-ucr-qualification.mjs", + "eval:ucr:qualification:gate": "node scripts/verify-ucr-qualification.mjs --require-qualified", + "evidence:ucr:study-design:update": "node scripts/verify-ucr-study-readiness.mjs --write", "evidence:ucr:update": "node scripts/verify-ucr-runtime.mjs --write", "eval:ucr:handoff:plan": "node scripts/run-ucr-handoff-smoke.mjs", "eval:ucr:handoff": "node scripts/run-ucr-handoff-smoke.mjs --execute", @@ -73,19 +77,29 @@ "eval:ucr:adapters": "node scripts/run-ucr-adapter-certification.mjs --write", "eval:ucr:compounding": "node scripts/run-ucr-compounding-study.mjs", "eval:ucr:production": "node scripts/run-ucr-production-study.mjs", + "eval:ucr:production:traffic:plan": "node scripts/assemble-ucr-production-traffic.mjs", + "eval:ucr:production:traffic": "node scripts/assemble-ucr-production-traffic.mjs --execute", "eval:ucr:context": "node scripts/run-mcp-context-audit.mjs", "evidence:ucr:assemble": "node scripts/assemble-ucr-evidence.mjs && node scripts/verify-ucr-runtime.mjs --write", "eval:ucr:multimodel:plan": "node scripts/run-ucr-multimodel-handoff.mjs", "eval:ucr:multimodel": "node scripts/run-ucr-multimodel-handoff.mjs --execute --directions all", "eval:ucr:effectiveness:plan": "node scripts/run-ucr-handoff-effectiveness.mjs", "eval:ucr:effectiveness": "node scripts/run-ucr-handoff-effectiveness.mjs --execute --pairs-per-direction 357", + "eval:ucr:full-study:plan": "node scripts/run-ucr-full-study.mjs", + "eval:ucr:full-study:powered-plan": "node scripts/run-ucr-full-study.mjs --powered", + "eval:ucr:identity:provision": "node scripts/provision-ucr-study-identity.mjs", + "eval:ucr:full-study:qualification": "node scripts/run-ucr-full-study.mjs --execute --runner-module scripts/ucr-study-driver-dispatch.mjs", + "eval:ucr:full-study": "node scripts/run-ucr-full-study.mjs --execute --powered --runner-module scripts/ucr-study-driver-dispatch.mjs", + "eval:ucr:competitive:plan": "node scripts/run-ucr-competitive-study.mjs", + "eval:ucr:competitive": "node scripts/run-ucr-competitive-study.mjs --execute", "eval:evidence": "node scripts/run-evidence-eval.mjs", "eval:evidence:plan": "node scripts/run-evidence-eval.mjs --dry-run", "eval:handoff": "node scripts/run-handoff-eval.mjs", "eval:handoff:plan": "node scripts/run-handoff-eval.mjs --dry-run", "eval:handoff:concurrent": "node scripts/run-concurrent-handoff-eval.mjs", - "verify:all": "npm run sync:hooks:check && npm run verify:clients && npm run verify:certification && npm run verify:harvest && npm run verify:ucr && npm run verify:live-graph && npm run verify:ui && npm run verify:interactions", + "verify:all": "npm run sync:hooks:check && npm run eval:ucr:qualification:verify && npm run verify:clients && npm run verify:certification && npm run verify:harvest && npm run verify:ucr && npm run verify:ucr:study-design && npm run verify:live-graph && npm run verify:ui && npm run verify:interactions", "verify:interactions": "node scripts/verify-wiki-interactions.mjs", + "dashboard:verify-live": "node scripts/verify-live-dashboard.mjs", "explore:wiki": "node scripts/explore-wiki.mjs" }, "repository": { diff --git a/scripts/assemble-ucr-evidence.mjs b/scripts/assemble-ucr-evidence.mjs index 43fb6d8fc..f9915ebfb 100644 --- a/scripts/assemble-ucr-evidence.mjs +++ b/scripts/assemble-ucr-evidence.mjs @@ -1,7 +1,13 @@ #!/usr/bin/env node import { generateKeyPairSync, randomBytes } from 'node:crypto'; -import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { + existsSync, + mkdirSync, + readFileSync, + readdirSync, + writeFileSync, +} from 'node:fs'; import { dirname, join } from 'node:path'; import { fileURLToPath } from 'node:url'; import { @@ -10,8 +16,10 @@ import { deriveReleaseMetrics, evidenceTierReport, releaseVerdict, + resolveEvidenceVerificationPublicKey, sealEvidenceLedger, sha256, + tieredReleaseVerdict, verifyEvidenceLedger, } from '../ucr/index.mjs'; @@ -19,6 +27,7 @@ const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); const RESULTS = join(ROOT, 'evals', 'ucr', 'results'); const OUTPUT = join(RESULTS, 'evidence-index-v2.json'); const definitions = [ + ['study-readiness', 'study-readiness-v1.json', 'conformance'], ['graph-scale', 'graph-scale-v1.json', 'conformance'], ['coordination-scale', 'coordination-scale-v1.json', 'conformance'], ['consolidation', 'consolidation-study-v1.json', 'conformance'], @@ -28,12 +37,12 @@ const definitions = [ ['production-exercise', 'production-exercise-v1.json', 'executable-smoke'], [ 'stateful-codex-to-claude', - 'stateful-preflight-codex-to-claude-v4.json', + 'stateful-matched-codex-to-claude-v1.json', 'executable-smoke', ], [ 'stateful-claude-to-codex', - 'stateful-preflight-claude-to-codex-v3.json', + 'stateful-matched-claude-to-codex-v1.json', 'executable-smoke', ], [ @@ -55,6 +64,24 @@ const definitions = [ false, ], ]; +for (const definition of [ + ['full-effectiveness', 'full-study-v1.json', 'effectiveness', true], + ['live-competitive', 'competitive-study-v1.json', 'superiority', true], + ['production-traffic', 'production-traffic-v1.json', 'production', true], +]) { + if (existsSync(join(RESULTS, definition[1]))) definitions.push(definition); +} +for (const filename of readdirSync(RESULTS) + .filter((name) => + /^full-study-(?:qualification|blocked)-.*\.json$/i.test(name) + ) + .sort()) + definitions.push([ + filename.replace(/\.json$/i, ''), + filename, + filename.includes('-blocked-') ? 'blocked-smoke' : 'executable-smoke', + false, + ]); function readReport([name, filename, evidenceClass, requiredPass = true]) { const path = join(RESULTS, filename); @@ -63,9 +90,35 @@ function readReport([name, filename, evidenceClass, requiredPass = true]) { const report = JSON.parse(readFileSync(path, 'utf8')); const { reportHash, ...body } = report; const reportHashValid = sha256(body) === reportHash; + const promotable = ['effectiveness', 'superiority', 'production'].includes( + evidenceClass + ); + let verificationPublicKey = null; + let keyResolutionError = null; + try { + const trustedKeyPath = + report.ledgerKeyId && /^[A-Za-z0-9._-]+$/.test(report.ledgerKeyId) + ? join( + ROOT, + 'evals', + 'ucr', + 'trusted-evidence-keys', + `${report.ledgerKeyId}.pem` + ) + : null; + verificationPublicKey = resolveEvidenceVerificationPublicKey(report, { + promotable, + trustedPublicKey: + trustedKeyPath && existsSync(trustedKeyPath) + ? readFileSync(trustedKeyPath, 'utf8') + : null, + }); + } catch (error) { + keyResolutionError = error instanceof Error ? error.message : String(error); + } const ledgerVerification = report.ledger ? verifyEvidenceLedger(report.ledger, { - publicKey: report.ledgerPublicKey || null, + publicKey: verificationPublicKey, }) : null; return { @@ -76,7 +129,9 @@ function readReport([name, filename, evidenceClass, requiredPass = true]) { present: true, valid: reportHashValid && + keyResolutionError === null && (!requiredPass || report.passed === true) && + (!promotable || ledgerVerification?.valid === true) && (!ledgerVerification || ledgerVerification.valid), reportHashValid, reportHash, @@ -86,6 +141,8 @@ function readReport([name, filename, evidenceClass, requiredPass = true]) { sourceTreeHash: report.sourceTreeHash || report.sourceHash || null, ledgerHash: ledgerVerification?.ledgerHash || null, ledgerVerification, + keyResolutionError, + verificationPublicKey, report, }; } @@ -144,12 +201,24 @@ const ledgerInputs = [ .filter((artifact) => artifact.valid && artifact.report?.ledger) .map((artifact) => ({ ledger: artifact.report.ledger, - publicKey: artifact.report.ledgerPublicKey || null, + publicKey: artifact.verificationPublicKey, })), ]; const derived = deriveReleaseMetrics(ledgerInputs); const tiers = evidenceTierReport(ledgerInputs); const verdict = releaseVerdict(derived.metrics); +const productionTraffic = artifacts.find( + (artifact) => artifact.name === 'production-traffic' +); +const studyReadiness = artifacts.find( + (artifact) => artifact.name === 'study-readiness' +); +const tieredVerdict = tieredReleaseVerdict(derived.metrics, { + production: + productionTraffic?.valid === true + ? productionTraffic.report?.readiness || null + : null, +}); const validArtifacts = artifacts.filter((artifact) => artifact.valid); const liveArtifacts = artifacts.filter( (artifact) => @@ -159,7 +228,8 @@ const liveArtifacts = artifacts.filter( const totalTraffic = (usage) => Number.isFinite(usage?.totalTokens) ? usage.totalTokens - : Number.isFinite(usage?.inputTokens) && Number.isFinite(usage?.outputTokens) + : Number.isFinite(usage?.inputTokens) && + Number.isFinite(usage?.outputTokens) ? usage.inputTokens + usage.outputTokens : null; const percentDelta = (control, runtime) => @@ -191,8 +261,7 @@ const liveDirectionMetrics = liveArtifacts.map((artifact) => { nativeGuardEnforced: row?.runtime?.nativeGuardEnforced ?? null, captureModelCalls: row?.runtime?.firstSuccessorCost?.additionalModelCalls ?? null, - consumerStaticSchemaTokens: - row?.runtime?.usage?.staticSchemaTokens ?? null, + consumerStaticSchemaTokens: row?.runtime?.usage?.staticSchemaTokens ?? null, capsuleTokens: row?.runtime?.usage?.capsuleTokens ?? null, controlTokenTraffic: controlTraffic, runtimeTokenTraffic: runtimeTraffic, @@ -238,7 +307,13 @@ const body = { 'metrics are derived only from ledgers at or above each metric evidence requirement', tiers, }, - artifacts: artifacts.map(({ report: _report, ...artifact }) => artifact), + artifacts: artifacts.map( + ({ + report: _report, + verificationPublicKey: _verificationPublicKey, + ...artifact + }) => artifact + ), summary: { artifactsValid: validArtifacts.length, artifactsTotal: artifacts.length, @@ -300,6 +375,18 @@ const body = { cognitiveReductionVsFull: artifacts.find( (artifact) => artifact.name === 'mcp-context' )?.report?.findings?.graphCaptureReductionVsFull, + studyDesign: studyReadiness?.report + ? { + passed: studyReadiness.report.passed, + trials: studyReadiness.report.trials, + providerInvocations: studyReadiness.report.providerInvocations, + mappedMetrics: studyReadiness.report.mappedMetrics, + representativeStudyClients: + studyReadiness.report.representativeStudyClients, + universalDriverClients: studyReadiness.report.universalDriverClients, + coverage: studyReadiness.report.coverage, + } + : null, }, conformanceLedger, conformancePublicKey, @@ -308,13 +395,14 @@ const body = { }), derived, verdict, + tieredVerdict, claims: { conformance: validArtifacts.length === artifacts.length, - executableCrossClient: - passedLiveDirections.length >= 2, + executableCrossClient: passedLiveDirections.length >= 2, effectiveness: tiers.effectiveness.status === 'present', superiority: tiers.superiority.status === 'present', production: tiers.production.status === 'present', + replacement: tieredVerdict.passed === true, }, }; const report = { ...body, reportHash: sha256(body) }; diff --git a/scripts/assemble-ucr-production-traffic.mjs b/scripts/assemble-ucr-production-traffic.mjs new file mode 100644 index 000000000..348912e69 --- /dev/null +++ b/scripts/assemble-ucr-production-traffic.mjs @@ -0,0 +1,207 @@ +#!/usr/bin/env node +/** Assemble privacy-safe real shadow/canary telemetry into production evidence. */ + +import { randomBytes } from 'node:crypto'; +import { existsSync, mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { dirname, isAbsolute, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { + PRODUCTION_TIMESTAMP_BUCKET_MS, + REQUIRED_TRAFFIC_STAGES, + canonicalJson, + createEvidenceRun, + loadProvisionedEvidenceIdentity, + productionReadiness, + productionTrafficReport, + pseudonymizeProductionSamples, + sealEvidenceLedger, + sha256, + sloReport, + verifyEvidenceLedger, + validateProductionSample, +} from '../ucr/index.mjs'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +const options = { + execute: false, + input: null, + releaseEvidence: join( + ROOT, + 'evals', + 'ucr', + 'results', + 'evidence-index-v2.json' + ), + faultEvidence: join( + ROOT, + 'evals', + 'ucr', + 'results', + 'production-exercise-v1.json' + ), + output: join(ROOT, 'evals', 'ucr', 'results', 'production-traffic-v1.json'), + rolloutStage: 'stable', + minimumSamples: 1000, + minimumDurationMs: 604800000, + minimumClients: 3, + minimumProjects: 3, +}; +for (let index = 2; index < process.argv.length; index++) { + const arg = process.argv[index]; + if (arg === '--execute') options.execute = true; + else if (arg.startsWith('--')) { + const key = arg + .slice(2) + .replace(/-([a-z])/g, (_, letter) => letter.toUpperCase()); + if (!(key in options)) throw new Error(`unknown option ${arg}`); + options[key] = process.argv[++index]; + } +} +for (const key of [ + 'minimumSamples', + 'minimumDurationMs', + 'minimumClients', + 'minimumProjects', +]) + options[key] = Number(options[key]); +for (const key of ['input', 'releaseEvidence', 'faultEvidence', 'output']) { + if (options[key]) + options[key] = isAbsolute(options[key]) + ? options[key] + : resolve(ROOT, options[key]); +} + +if (!options.execute) { + console.log( + JSON.stringify( + { + execute: false, + requiredStages: REQUIRED_TRAFFIC_STAGES, + thresholds: { + minimumSamples: options.minimumSamples, + minimumDurationMs: options.minimumDurationMs, + minimumClients: options.minimumClients, + minimumProjects: options.minimumProjects, + }, + requiredFields: [ + 'timestamp', + 'realTraffic=true', + 'optIn=true', + 'rolloutStage', + 'client', + 'projectId', + 'available', + 'latencyMs', + 'contextOverhead', + 'correctnessDelta', + 'severeHarm', + 'unauthorizedAccess', + 'privacyViolation', + ], + prohibitedFields: ['prompt', 'transcript', 'rawOutput'], + note: 'Local exercises are never accepted as production traffic.', + }, + null, + 2 + ) + ); + process.exit(0); +} +if (!options.input || !existsSync(options.input)) + throw new Error('--input must identify an existing real-traffic JSONL file'); +for (const path of [options.releaseEvidence, options.faultEvidence]) + if (!existsSync(path)) + throw new Error(`required evidence is missing: ${path}`); +const signingIdentity = loadProvisionedEvidenceIdentity(); +const pseudonymSecret = process.env.UCR_TRAFFIC_PSEUDONYM_SECRET; +const pseudonymKeyId = String( + process.env.UCR_TRAFFIC_PSEUDONYM_KEY_ID || '' +).trim(); +if (!pseudonymSecret || !pseudonymKeyId) + throw new Error( + 'UCR_TRAFFIC_PSEUDONYM_SECRET and UCR_TRAFFIC_PSEUDONYM_KEY_ID are required' + ); + +const rawSamples = readFileSync(options.input, 'utf8') + .split(/\r?\n/) + .filter(Boolean) + .map((line, index) => { + const sample = JSON.parse(line); + const validation = validateProductionSample(sample); + if (!validation.valid) + throw new Error( + `production sample ${index} is invalid: ${validation.diagnostics.join('; ')}` + ); + return sample; + }); +const samples = pseudonymizeProductionSamples(rawSamples, { + secret: pseudonymSecret, + keyId: pseudonymKeyId, + timestampBucketMs: PRODUCTION_TIMESTAMP_BUCKET_MS, +}); +const releaseEvidence = JSON.parse( + readFileSync(options.releaseEvidence, 'utf8') +); +const faultEvidence = JSON.parse(readFileSync(options.faultEvidence, 'utf8')); +const traffic = productionTrafficReport(samples, { + minimumSamples: options.minimumSamples, + minimumDurationMs: options.minimumDurationMs, + minimumClients: options.minimumClients, + minimumProjects: options.minimumProjects, +}); +const slos = sloReport(samples); +const sourceTreeHash = sha256([ + readFileSync(fileURLToPath(import.meta.url), 'utf8'), + readFileSync(join(ROOT, 'ucr', 'rollout.mjs'), 'utf8'), + releaseEvidence.reportHash, + faultEvidence.reportHash, +]); +const run = createEvidenceRun({ + runId: `production-traffic-${randomBytes(8).toString('hex')}`, + evidenceClass: 'production', + benchmarkHash: + releaseEvidence.derived?.derivedHash || releaseEvidence.reportHash, + sourceTreeHash, + runner: { name: 'ucr-production-traffic', node: process.version }, +}); +const ledger = sealEvidenceLedger(run, samples, { + privateKey: signingIdentity.privateKey, +}); +const ledgerVerification = verifyEvidenceLedger(ledger, { + publicKey: signingIdentity.publicKey, +}); +const readiness = productionReadiness({ + release: releaseEvidence.verdict, + evidenceClasses: ['effectiveness', 'superiority', 'production'], + slos, + faults: faultEvidence.faults, + recovery: faultEvidence.recovery, + traffic, + rolloutStage: options.rolloutStage, +}); +const body = { + schemaVersion: 'ucr.production-traffic/1', + executedAt: new Date().toISOString(), + sourceTreeHash, + traffic, + slos, + readiness, + ledger, + ledgerKeyId: signingIdentity.keyId, + identifiersPseudonymized: true, + pseudonymizationKeyId: pseudonymKeyId, + ledgerVerification, + passed: readiness.ready && ledgerVerification.valid, +}; +const report = { ...body, reportHash: sha256(body) }; +mkdirSync(dirname(options.output), { recursive: true }); +writeFileSync(options.output, `${JSON.stringify(report, null, 2)}\n`, 'utf8'); +console.log( + canonicalJson({ + output: options.output, + samples: traffic.metrics.samples, + readiness: readiness.status, + ledgerHash: ledger.ledgerHash, + }) +); +if (!report.passed) process.exitCode = 1; diff --git a/scripts/explore-wiki.mjs b/scripts/explore-wiki.mjs index 01b70346d..b30670f93 100644 --- a/scripts/explore-wiki.mjs +++ b/scripts/explore-wiki.mjs @@ -5,7 +5,7 @@ * verify-wiki-interactions.mjs asserts things I already expected to be true. * This does the opposite: it performs awkward, impatient, out-of-order things a * real person does -- double-clicking, typing while a render is mid-flight, - * switching modes repeatedly, resizing during physics, reloading after a + * switching modes repeatedly, resizing the 3D scene, reloading after a * mutation, retiring findings until none are left -- and REPORTS anomalies * rather than checking for known answers. * @@ -38,13 +38,22 @@ const drawerOpen = (page) => page.locator('#wiki-detail').isVisible(); async function main() { mkdirSync(SHOTS, { recursive: true }); const browser = await chromium.launch(); - const page = await browser.newPage({ viewport: { width: 1440, height: 900 } }); + const page = await browser.newPage({ + viewport: { width: 1440, height: 900 }, + }); const errors = []; - page.on('console', (m) => { if (m.type() === 'error') errors.push(m.text()); }); + page.on('console', (m) => { + if (m.type() === 'error') errors.push(m.text()); + }); page.on('pageerror', (e) => errors.push(`pageerror: ${e.message}`)); - page.on('requestfailed', (r) => errors.push(`requestfailed: ${r.url()} ${r.failure()?.errorText}`)); - page.on('dialog', async (d) => { note('dialog', `unexpected: ${d.message()}`); await d.dismiss(); }); + page.on('requestfailed', (r) => + errors.push(`requestfailed: ${r.url()} ${r.failure()?.errorText}`) + ); + page.on('dialog', async (d) => { + note('dialog', `unexpected: ${d.message()}`); + await d.dismiss(); + }); const drain = (where) => { while (errors.length) note(where, errors.shift()); @@ -61,9 +70,12 @@ async function main() { await page.waitForTimeout(900); drain('triple-click list item'); if (!(await drawerOpen(page))) note('triple-click', 'drawer did not open'); - const centres = await page.locator('#wiki-graph .wiki-node.is-center').count(); - if (centres !== 1) note('triple-click', `${centres} centre nodes after rapid clicks`); - else ok('rapid clicking leaves exactly one centre'); + const selected = await page + .locator('#wiki-graph-3d') + .getAttribute('data-selected'); + if (!selected) + note('triple-click', 'rapid selection left no selected 3D node'); + else ok('rapid clicking leaves one selected 3D node'); /* --- 2. Mode toggle hammered ------------------------------------------ */ for (let i = 0; i < 6; i++) { @@ -73,11 +85,13 @@ async function main() { await page.waitForTimeout(1500); drain('mode hammering'); const nodesAfter = await page.locator('#wiki-graph .wiki-node').count(); - if (nodesAfter === 0) note('mode hammering', 'graph is empty after rapid mode switching'); + if (nodesAfter === 0) + note('mode hammering', 'graph is empty after rapid mode switching'); else ok(`graph survives rapid mode switching (${nodesAfter} nodes)`); const activeModes = await page.locator('.wiki-mode.is-active').count(); - if (activeModes !== 1) note('mode hammering', `${activeModes} modes marked active`); + if (activeModes !== 1) + note('mode hammering', `${activeModes} modes marked active`); else ok('exactly one mode is active'); /* --- 3. Typing while a render is in flight ---------------------------- */ @@ -88,7 +102,11 @@ async function main() { await page.waitForTimeout(1600); drain('type during render'); const afterRace = await listCount(page); - if (afterRace === 0) note('type during render', 'list empty after search race with an empty query'); + if (afterRace === 0) + note( + 'type during render', + 'list empty after search race with an empty query' + ); else ok(`list intact after search/render race (${afterRace})`); /* --- 4. Fast successive searches (debounce race) ---------------------- */ @@ -101,7 +119,10 @@ async function main() { const settled = await listCount(page); const reported = await page.locator('#wiki-count').innerText(); if (!reported.startsWith(String(settled))) { - note('debounce race', `list shows ${settled} but counter says "${reported}"`); + note( + 'debounce race', + `list shows ${settled} but counter says "${reported}"` + ); } else ok(`search settles consistently (${reported})`); /* --- 5. Keyboard-only traversal --------------------------------------- */ @@ -122,7 +143,12 @@ async function main() { if (id) reachable.add(String(id)); } drain('keyboard traversal'); - for (const needed of ['wiki-search', 'wiki-type', 'mode-focus', 'mode-constellation']) { + for (const needed of [ + 'wiki-search', + 'wiki-type', + 'mode-focus', + 'mode-constellation', + ]) { if (![...reachable].some((r) => r.includes(needed))) { note('keyboard', `${needed} is not reachable by Tab`); } @@ -130,56 +156,96 @@ async function main() { ok(`Tab reaches ${reachable.size} distinct controls`); /* --- 6. Walk every node in the focus graph ---------------------------- */ - // Deliberately begin this walk from CONSTELLATION, because that is the state - // that exposed the mode/view desync: selecting a node rendered a focus graph - // while the toggle still read Constellation, and the next re-render discarded - // the selection. + // Deliberately begin this walk from 3D. Selection must remain in 3D until the + // user explicitly asks for the bounded one-hop Focus view. await page.locator('#mode-constellation').click(); await page.waitForTimeout(1200); await page.locator('#wiki-list li').first().click(); + const modesAfterSelect = await page + .locator('.wiki-mode.is-active') + .evaluateAll((modes) => modes.map((mode) => mode.id)); + if ( + modesAfterSelect.length !== 1 || + modesAfterSelect[0] !== 'mode-constellation' + ) { + note( + 'mode desync', + `selecting a node left ${JSON.stringify(modesAfterSelect)} marked active` + ); + } else ok('selecting a node preserves the 3D view'); + await page.locator('#mode-focus').click(); await page.waitForSelector('#wiki-graph .wiki-node.is-center'); - const modeAfterSelect = await page.locator('.wiki-mode.is-active').getAttribute('id'); - if (modeAfterSelect !== 'mode-focus') { - note('mode desync', `selecting a node left "${modeAfterSelect}" marked active`); - } else ok('selecting a node switches the toggle to Focus'); let walked = 0; for (let hop = 0; hop < 8; hop++) { - const next = await page.evaluate(() => - document.querySelector('#wiki-graph .wiki-node:not(.is-center)')?.dataset.id || null); + const next = await page.evaluate( + () => + document.querySelector('#wiki-graph .wiki-node:not(.is-center)') + ?.dataset.id || null + ); if (!next) break; await page.locator(`#wiki-graph .wiki-node[data-id="${next}"]`).click(); await page.waitForTimeout(500); - const centre = await page.evaluate(() => - document.querySelector('#wiki-graph .wiki-node.is-center')?.dataset.id || null); - if (centre !== next) { note('graph walk', `clicked ${next} but centre is ${centre}`); break; } + const centre = await page.evaluate( + () => + document.querySelector('#wiki-graph .wiki-node.is-center')?.dataset + .id || null + ); + if (centre !== next) { + note('graph walk', `clicked ${next} but centre is ${centre}`); + break; + } walked++; } drain('graph walk'); ok(`walked ${walked} hops through the graph without desync`); - /* --- 7. Resize during constellation physics --------------------------- */ + /* --- 7. Resize the 3D constellation ------------------------------------ */ await page.locator('#mode-constellation').click(); await page.setViewportSize({ width: 900, height: 700 }); await page.waitForTimeout(300); await page.setViewportSize({ width: 1440, height: 900 }); await page.waitForTimeout(1500); - drain('resize during physics'); - const spread = await page.evaluate(() => { - const boxes = [...document.querySelectorAll('#wiki-graph .wiki-node circle')].map((c) => c.getBoundingClientRect()); - if (boxes.length < 2) return -1; - const xs = boxes.map((b) => b.x); - return Math.round(Math.max(...xs) - Math.min(...xs)); + drain('resize 3D scene'); + const projection = await page.evaluate(() => { + const host = document.getElementById('wiki-graph-3d'); + if (!host) return { hostPresent: false, spread: null }; + const nodes = host.__knowledgeGraph3d?.projectedNodes() || []; + if (nodes.length < 2) return { hostPresent: true, spread: -1 }; + const xs = nodes.map((node) => node.x); + return { + hostPresent: true, + spread: Math.round(Math.max(...xs) - Math.min(...xs)), + }; }); - if (spread === 0) note('resize during physics', 'all nodes collapsed to one point'); - else ok(`layout survives resize mid-physics (${spread}px spread)`); + if (!projection.hostPresent) + note('resize 3D scene', '3D graph host is missing after resize'); + else if (projection.spread === -1) + note( + 'resize 3D scene', + 'fewer than two projected nodes remain after resize' + ); + else if (projection.spread <= 0) + note('resize 3D scene', 'all nodes collapsed to one point'); + else ok(`3D layout survives resize (${projection.spread}px spread)`); const clipped = await page.evaluate(() => { - const svg = document.getElementById('wiki-graph').getBoundingClientRect(); - return [...document.querySelectorAll('#wiki-graph .wiki-node text')] - .filter((t) => { const b = t.getBoundingClientRect(); return b.right > svg.right + 2 || b.left < svg.left - 2; }).length; + const host = document.getElementById('wiki-graph-3d'); + if (!host) return null; + const box = host.getBoundingClientRect(); + const nodes = host.__knowledgeGraph3d?.projectedNodes() || []; + return nodes.filter( + (node) => + node.x < 0 || node.y < 0 || node.x > box.width || node.y > box.height + ).length; }); - if (clipped) note('resize during physics', `${clipped} labels clipped after resize`); - else ok('no labels clipped after resize'); + if (clipped === null) + note( + 'resize 3D scene', + 'cannot check clipping because the 3D host is missing' + ); + else if (clipped) + note('resize 3D scene', `${clipped} projected nodes clipped after resize`); + else ok('no projected nodes clipped after resize'); /* --- 8. Tab switching with a drawer open ------------------------------ */ await page.locator('#mode-focus').click(); @@ -190,20 +256,27 @@ async function main() { drain('tab switch with drawer open'); if (await drawerOpen(page)) { const exploreHidden = await page.locator('#tab-explore').isHidden(); - if (exploreHidden) note('tab switch', 'drawer still shows an Explore selection while the Audit tab is active'); + if (exploreHidden) + note( + 'tab switch', + 'drawer still shows an Explore selection while the Audit tab is active' + ); } await page.locator('.wiki-tab[data-tab="explore"]').click(); await page.waitForTimeout(400); /* --- 9. Zoom ----------------------------------------------------------- */ for (const zoom of [2, 0.5, 1]) { - await page.evaluate((z) => { document.body.style.zoom = String(z); }, zoom); + await page.evaluate((z) => { + document.body.style.zoom = String(z); + }, zoom); await page.waitForTimeout(600); const overflow = await page.evaluate(() => { const el = document.scrollingElement; return el.scrollWidth - el.clientWidth; }); - if (overflow > 2) note('zoom', `horizontal overflow of ${overflow}px at zoom ${zoom}`); + if (overflow > 2) + note('zoom', `horizontal overflow of ${overflow}px at zoom ${zoom}`); else ok(`no horizontal overflow at zoom ${zoom}`); } drain('zoom'); @@ -219,27 +292,37 @@ async function main() { const afterReload = await listCount(page); drain('reload after mutation'); if (afterReload !== beforeReload) { - note('reload after mutation', `list was ${beforeReload}, is ${afterReload} after reload`); + note( + 'reload after mutation', + `list was ${beforeReload}, is ${afterReload} after reload` + ); } else ok(`state survives reload (${afterReload} findings)`); /* --- 11. Chained corrections ------------------------------------------- */ for (let round = 0; round < 3; round++) { await page.locator('#wiki-list li').first().click(); await page.waitForSelector('#wiki-detail:visible'); - await page.locator('#detail-claim').fill(`chained correction round ${round}`); + await page + .locator('#detail-claim') + .fill(`chained correction round ${round}`); await page.locator('#detail-correct').click(); await page.waitForTimeout(800); drain(`correction round ${round}`); } - const chained = await page.locator('#wiki-list li', { hasText: 'chained correction round 2' }).count(); - if (!chained) note('chained corrections', 'the final correction is not in the list'); + const chained = await page + .locator('#wiki-list li', { hasText: 'chained correction round 2' }) + .count(); + if (!chained) + note('chained corrections', 'the final correction is not in the list'); else ok('corrections can be chained'); /* --- 12. Retire everything, then look at the empty state ---------------- */ let guard = 0; - while (await listCount(page) > 0 && guard++ < 40) { + while ((await listCount(page)) > 0 && guard++ < 40) { await page.locator('#wiki-list li').first().click(); - await page.waitForSelector('#wiki-detail:visible', { timeout: 4000 }).catch(() => {}); + await page + .waitForSelector('#wiki-detail:visible', { timeout: 4000 }) + .catch(() => {}); const retire = page.locator('#detail-retire'); if (!(await retire.count())) break; await retire.click(); @@ -258,26 +341,43 @@ async function main() { return s.total; }); const stats = await page.locator('#graph-stats').innerText(); - const balance = await page.locator('#balance-verdict').innerText().catch(() => ''); - const auditText = await page.evaluate(() => { - document.querySelector('.wiki-tab[data-tab="audit"]').click(); - return new Promise((r) => setTimeout(() => r(document.getElementById('audit-groups').innerText), 700)); - }); + const balance = await page + .locator('#balance-verdict') + .innerText() + .catch(() => ''); + await page + .locator('.wiki-tab[data-tab="audit"]') + .click() + .catch(() => note('empty state', 'audit tab is missing')); + await page.waitForTimeout(700); + const auditText = await page + .locator('#audit-groups') + .innerText() + .catch(() => ''); + if (!auditText) note('empty state', 'audit groups are missing or empty'); if (emptyIndexNull === 0) { if (!/0 findings|No graph|healthy|Nothing/i.test(`${stats} ${auditText}`)) { - note('empty state', `graph is empty but UI says "${stats.trim()}" / audit "${auditText.slice(0, 60)}"`); + note( + 'empty state', + `graph is empty but UI says "${stats.trim()}" / audit "${auditText.slice(0, 60)}"` + ); } else ok('empty state reads sensibly'); } else ok(`${emptyIndexNull} findings remain (nothing fully emptied)`); if (balance) ok(`balance still reports: ${balance.slice(0, 70)}`); - await page.screenshot({ path: join(SHOTS, 'after-exploration.png'), fullPage: true }); + await page.screenshot({ + path: join(SHOTS, 'after-exploration.png'), + fullPage: true, + }); drain('final'); await browser.close(); - console.log(`\n${anomalies.length} anomal${anomalies.length === 1 ? 'y' : 'ies'} found`); + console.log( + `\n${anomalies.length} anomal${anomalies.length === 1 ? 'y' : 'ies'} found` + ); for (const a of anomalies) console.log(` [${a.where}] ${a.what}`); console.log(`screenshots: ${SHOTS}`); } diff --git a/scripts/postinstall.cjs b/scripts/postinstall.cjs index 328a58e8d..f0ac82598 100644 --- a/scripts/postinstall.cjs +++ b/scripts/postinstall.cjs @@ -10,15 +10,20 @@ const path = require('path'); const fs = require('fs'); // Detect if we're in a CI environment or non-interactive shell -const isCI = process.env.CI === 'true' || - process.env.CONTINUOUS_INTEGRATION === 'true' || - process.env.GITHUB_ACTIONS === 'true' || - !process.stdout.isTTY; +const isCI = + process.env.CI === 'true' || + process.env.CONTINUOUS_INTEGRATION === 'true' || + process.env.GITHUB_ACTIONS === 'true' || + !process.stdout.isTTY; // Skip hook installation in CI or when installing as a dependency if (isCI || process.env.npm_config_global !== 'true') { - console.log('[token-optimizer-mcp] Skipping automatic hook installation (CI or local install)'); - console.log('[token-optimizer-mcp] To install hooks manually, see: https://github.com/ooples/token-optimizer-mcp#installation'); + console.log( + '[token-optimizer-mcp] Skipping automatic hook installation (CI or local install)' + ); + console.log( + '[token-optimizer-mcp] To install hooks manually, see: https://github.com/ooples/token-optimizer-mcp#installation' + ); process.exit(0); } @@ -38,8 +43,12 @@ try { try { execSync('powershell -Command "exit 0"', { stdio: 'ignore' }); } catch (error) { - console.warn('[token-optimizer-mcp] PowerShell not available, skipping hook installation'); - console.log('[token-optimizer-mcp] Run install-hooks.ps1 manually to enable hooks'); + console.warn( + '[token-optimizer-mcp] PowerShell not available, skipping hook installation' + ); + console.log( + '[token-optimizer-mcp] Run install-hooks.ps1 manually to enable hooks' + ); process.exit(0); } @@ -51,7 +60,10 @@ try { try { fs.chmodSync(installScript, 0o755); } catch (error) { - console.warn('[token-optimizer-mcp] Could not make install script executable:', error.message); + console.warn( + '[token-optimizer-mcp] Could not make install script executable:', + error.message + ); } command = `bash "${installScript}"`; @@ -59,7 +71,10 @@ try { // Check if install script exists if (!fs.existsSync(installScript)) { - console.warn('[token-optimizer-mcp] Install script not found:', installScript); + console.warn( + '[token-optimizer-mcp] Install script not found:', + installScript + ); console.log('[token-optimizer-mcp] Skipping automatic hook installation'); process.exit(0); } @@ -69,18 +84,26 @@ try { // Run the installer execSync(command, { stdio: 'inherit', - cwd: packageRoot + cwd: packageRoot, }); console.log('[token-optimizer-mcp] ✓ Hooks installed successfully!'); - console.log('[token-optimizer-mcp] Token optimization is now active for all Claude Code operations'); - + console.log( + '[token-optimizer-mcp] Token optimization is active for supported coding agents and CLI tools' + ); } catch (error) { - console.warn('[token-optimizer-mcp] Hook installation encountered an issue:', error.message); - console.log('[token-optimizer-mcp] You can manually install hooks by running:'); + console.warn( + '[token-optimizer-mcp] Hook installation encountered an issue:', + error.message + ); + console.log( + '[token-optimizer-mcp] You can manually install hooks by running:' + ); if (process.platform === 'win32') { - console.log('[token-optimizer-mcp] powershell -ExecutionPolicy Bypass -File install-hooks.ps1'); + console.log( + '[token-optimizer-mcp] powershell -ExecutionPolicy Bypass -File install-hooks.ps1' + ); } else { console.log('[token-optimizer-mcp] bash install-hooks.sh'); } diff --git a/scripts/provision-ucr-study-identity.mjs b/scripts/provision-ucr-study-identity.mjs new file mode 100644 index 000000000..d0ae461dd --- /dev/null +++ b/scripts/provision-ucr-study-identity.mjs @@ -0,0 +1,78 @@ +#!/usr/bin/env node +/** Provision operator-owned study signing and hidden-variant material. */ + +import { + createPrivateKey, + createPublicKey, + generateKeyPairSync, + randomBytes, + sign, + verify, +} from 'node:crypto'; +import { + chmodSync, + existsSync, + mkdirSync, + readFileSync, + writeFileSync, +} from 'node:fs'; +import { homedir } from 'node:os'; +import { isAbsolute, join, resolve } from 'node:path'; +import { canonicalJson, sha256 } from '../ucr/index.mjs'; + +const configured = + process.argv[2] || + process.env.TOKEN_OPTIMIZER_UCR_IDENTITY_DIR || + join(homedir(), '.token-optimizer-mcp', 'ucr-study-identity'); +const root = isAbsolute(configured) ? configured : resolve(configured); +const privateKeyPath = join(root, 'evidence-private.pem'); +const publicKeyPath = join(root, 'evidence-public.pem'); +const studySecretPath = join(root, 'study-secret.txt'); +mkdirSync(root, { recursive: true }); + +const present = [privateKeyPath, publicKeyPath, studySecretPath].map( + existsSync +); +if (present.some(Boolean) && !present.every(Boolean)) + throw new Error( + 'refusing to replace a partial study identity; restore or remove the named files explicitly' + ); +if (!present.every(Boolean)) { + const { privateKey, publicKey } = generateKeyPairSync('ed25519'); + writeFileSync( + privateKeyPath, + privateKey.export({ type: 'pkcs8', format: 'pem' }), + { mode: 0o600 } + ); + writeFileSync( + publicKeyPath, + publicKey.export({ type: 'spki', format: 'pem' }), + { mode: 0o600 } + ); + writeFileSync(studySecretPath, randomBytes(48).toString('base64url'), { + mode: 0o600, + }); +} +for (const path of [privateKeyPath, publicKeyPath, studySecretPath]) + chmodSync(path, 0o600); + +const privateKey = createPrivateKey(readFileSync(privateKeyPath, 'utf8')); +const publicKeyText = readFileSync(publicKeyPath, 'utf8'); +const publicKey = createPublicKey(publicKeyText); +const challenge = randomBytes(32); +const signature = sign(null, challenge, privateKey); +if (!verify(null, challenge, publicKey, signature)) + throw new Error('provisioned study identity has a mismatched keypair'); +const studySecret = readFileSync(studySecretPath, 'utf8').trim(); +if (studySecret.length < 32) + throw new Error('provisioned hidden-variant secret is too short'); +process.stdout.write( + canonicalJson({ + root, + keyId: `ucr-ed25519-${sha256(publicKeyText).slice(0, 20)}`, + privateKeyPath, + publicKeyPath, + studySecretPath, + created: !present.every(Boolean), + }) +); diff --git a/scripts/run-ucr-competitive-study.mjs b/scripts/run-ucr-competitive-study.mjs new file mode 100644 index 000000000..120c4f37a --- /dev/null +++ b/scripts/run-ucr-competitive-study.mjs @@ -0,0 +1,447 @@ +#!/usr/bin/env node +/** Fair live-product comparison using a completed effectiveness trial set. */ + +import { spawnSync } from 'node:child_process'; +import { randomBytes } from 'node:crypto'; +import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, isAbsolute, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { + PRODUCT_BASELINE_KINDS, + REQUIRED_COMPETITIVE_BASELINES, + bootstrapPaired, + buildFullStudyPlan, + canonicalJson, + createEvidenceRun, + freezeBenchmark, + gradeStudyFixture, + loadProvisionedEvidenceIdentity, + materializeStudyFixture, + paretoFront, + preRegisterBenchmark, + sealEvidenceLedger, + sha256, + validateCompetitiveEvidence, + validateFairRun, + verifyEvidenceLedger, +} from '../ucr/index.mjs'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +const options = { + execute: false, + effectivenessReport: join( + ROOT, + 'evals', + 'ucr', + 'results', + 'full-study-v1.json' + ), + products: join(ROOT, 'evals', 'ucr', 'competitors-v1.json'), + benchmark: join(ROOT, 'evals', 'ucr', 'benchmark-v1.json'), + runnerCommand: process.env.UCR_COMPETITOR_RUNNER || null, + repetitions: 2, + timeoutMs: 600_000, + output: join(ROOT, 'evals', 'ucr', 'results', 'competitive-study-v1.json'), +}; +for (let index = 2; index < process.argv.length; index++) { + const arg = process.argv[index]; + if (arg === '--execute') options.execute = true; + else if (arg.startsWith('--')) { + const key = arg + .slice(2) + .replace(/-([a-z])/g, (_, letter) => letter.toUpperCase()); + if (!(key in options)) throw new Error(`unknown option ${arg}`); + options[key] = process.argv[++index]; + } +} +options.repetitions = Number(options.repetitions); +options.timeoutMs = Number(options.timeoutMs); +if (!Number.isInteger(options.timeoutMs) || options.timeoutMs <= 0) + throw new Error('--timeout-ms must be a positive integer'); +for (const key of ['effectivenessReport', 'products', 'benchmark', 'output']) + options[key] = isAbsolute(options[key]) + ? options[key] + : resolve(ROOT, options[key]); + +const registry = JSON.parse(readFileSync(options.products, 'utf8')); +const baselines = registry.baselines || []; +const kinds = new Set(baselines.map((baseline) => baseline.kind)); +const missingKinds = REQUIRED_COMPETITIVE_BASELINES.filter( + (kind) => !kinds.has(kind) +); +const productReady = baselines + .filter((baseline) => REQUIRED_COMPETITIVE_BASELINES.includes(baseline.kind)) + .every( + (baseline) => + baseline.liveExecution === true && + baseline.version && + baseline.configuration && + (!PRODUCT_BASELINE_KINDS.includes(baseline.kind) || + baseline.namedProduct === true) + ); + +if (!options.execute) { + console.log( + JSON.stringify( + { + execute: false, + requiredBaselines: REQUIRED_COMPETITIVE_BASELINES, + registeredBaselines: [...kinds], + missingKinds, + productReady, + repetitions: options.repetitions, + fairnessFields: [ + 'model', + 'modelVersion', + 'taskId', + 'permissionsHash', + 'contextBudget', + 'retryBudget', + 'toolBudget', + ], + note: productReady + ? 'Registry can execute a superiority study.' + : 'Reference algorithms remain useful controls but cannot support a product-superiority claim.', + }, + null, + 2 + ) + ); + process.exit(missingKinds.length ? 1 : 0); +} + +if (!existsSync(options.effectivenessReport)) + throw new Error('a completed effectiveness report is required'); +if (!options.runnerCommand || !existsSync(options.runnerCommand)) + throw new Error('UCR_COMPETITOR_RUNNER must identify an executable wrapper'); +if (missingKinds.length) + throw new Error( + `competitive registry is missing: ${missingKinds.join(', ')}` + ); +if (!productReady) + throw new Error('reference-only baselines cannot be promoted to superiority'); +if (!Number.isInteger(options.repetitions) || options.repetitions < 2) + throw new Error('competitive reproduction requires at least two repetitions'); +if (!process.env.UCR_STUDY_SECRET) + throw new Error( + 'UCR_STUDY_SECRET is required to reproduce the frozen hidden variants' + ); +const signingIdentity = loadProvisionedEvidenceIdentity(); + +const effectiveness = JSON.parse( + readFileSync(options.effectivenessReport, 'utf8') +); +if ( + effectiveness.evidenceClass !== 'effectiveness' || + effectiveness.complete !== true || + effectiveness.ledgerKeyId !== signingIdentity.keyId || + !effectiveness.ledger?.run || + !verifyEvidenceLedger(effectiveness.ledger, { + publicKey: signingIdentity.publicKey, + }).valid +) + throw new Error( + 'competitive execution requires complete signed effectiveness evidence' + ); +const benchmarkSource = JSON.parse(readFileSync(options.benchmark, 'utf8')); +const benchmark = freezeBenchmark(benchmarkSource); +if (benchmark.manifestHash !== effectiveness.ledger.run.benchmarkHash) + throw new Error('effectiveness and competitive benchmark hashes differ'); +const taskById = new Map(benchmark.tasks.map((task) => [task.id, task])); +const registration = preRegisterBenchmark( + benchmarkSource, + benchmarkSource.pilot +); +const reproducedPlan = buildFullStudyPlan({ + benchmark, + clients: effectiveness.plan.clients, + secret: process.env.UCR_STUDY_SECRET, + registration, + pairedRepetitionsPerCell: effectiveness.plan.pairedRepetitionsPerCell, + negativeRepetitionsPerCell: effectiveness.plan.negativeRepetitionsPerCell, + includeSameClient: effectiveness.plan.includeSameClient, + budgets: effectiveness.plan.budgets, + minimumSubgroupPairs: effectiveness.plan.minimumSubgroupPairs, +}); +if (reproducedPlan.planHash !== effectiveness.plan.planHash) + throw new Error( + 'hidden study plan cannot be reproduced from the frozen secret' + ); +const trialById = new Map( + reproducedPlan.trials.map((trial) => [trial.trialId, trial]) +); +const ucrRows = effectiveness.ledger.rows.filter( + (row) => row.arm === 'runtime' +); +const raw = []; +const studyRoot = mkdtempSync(join(tmpdir(), 'ucr-competitive-study-')); + +try { + for (const baseline of baselines.filter((candidate) => + REQUIRED_COMPETITIVE_BASELINES.includes(candidate.kind) + )) { + for (const ucrRow of ucrRows) { + const task = taskById.get(ucrRow.taskId); + const plannedTrial = trialById.get(ucrRow.trialId); + if (!task || !plannedTrial) + throw new Error(`missing frozen task binding for ${ucrRow.trialId}`); + const reference = { + model: ucrRow.model, + modelVersion: ucrRow.modelVersion, + taskId: ucrRow.taskId, + permissionsHash: ucrRow.permissionsHash, + contextBudget: ucrRow.contextBudget, + retryBudget: ucrRow.retryBudget, + toolBudget: ucrRow.toolBudget, + }; + for (let repetition = 0; repetition < options.repetitions; repetition++) { + const fixture = materializeStudyFixture({ + task, + trial: plannedTrial, + root: join( + studyRoot, + sha256(`${baseline.kind}:${ucrRow.trialId}:${repetition}`).slice( + 0, + 24 + ) + ), + }); + const publicTask = { ...task }; + delete publicTask.grader; + delete publicTask.privateGrader; + delete publicTask.hiddenAnswer; + publicTask.prompt = plannedTrial.variantPrompt; + publicTask.publicVariant = plannedTrial.publicVariant; + const request = { + schemaVersion: 'ucr.competitive-trial-request/1', + baseline, + reference, + trial: { + pairId: ucrRow.pairId, + taskId: ucrRow.taskId, + family: ucrRow.family, + direction: ucrRow.direction, + repetition, + }, + task: publicTask, + fixture: fixture.public, + }; + const child = spawnSync(options.runnerCommand, [], { + cwd: fixture.workspace, + input: JSON.stringify(request), + encoding: 'utf8', + timeout: options.timeoutMs, + maxBuffer: 64 * 1024 * 1024, + windowsHide: true, + shell: false, + }); + let result = null; + try { + result = JSON.parse(child.stdout || 'null'); + } catch { + // Invalid output is preserved as a hashed failed result below. + } + const grade = gradeStudyFixture({ + task, + fixture, + actionAudit: Array.isArray(result?.actionAudit) + ? result.actionAudit + : [], + }); + const fairness = validateFairRun(result || {}, reference); + raw.push({ + baselineKind: baseline.kind, + baselineName: baseline.name, + namedProduct: baseline.namedProduct === true, + pairId: ucrRow.pairId, + taskId: ucrRow.taskId, + family: ucrRow.family, + direction: ucrRow.direction, + repetition, + fair: fairness.fair, + fairnessMismatches: fairness.mismatches, + correct: grade.correct, + severeHarm: grade.severeHarm, + totalTokens: result?.totalTokens ?? null, + latencyMs: result?.latencyMs ?? null, + exitCode: child.status, + liveExecution: result?.liveExecution === true, + versionPinned: result?.version === baseline.version, + configurationPublished: + result?.configurationHash === sha256(baseline.configuration), + independentGrade: grade.proseUsedAsOracle === false, + changedProtected: grade.changedProtected, + outputHash: sha256(String(child.stdout || '')), + errorHash: sha256(String(child.stderr || child.error?.message || '')), + }); + rmSync(fixture.workspace, { + recursive: true, + force: true, + maxRetries: 10, + retryDelay: 100, + }); + } + } + } +} finally { + rmSync(studyRoot, { + recursive: true, + force: true, + maxRetries: 10, + retryDelay: 100, + }); +} + +const summaryRows = []; +for (const baseline of baselines.filter((candidate) => + REQUIRED_COMPETITIVE_BASELINES.includes(candidate.kind) +)) { + const runs = raw.filter((row) => row.baselineKind === baseline.kind); + const primary = runs.filter((row) => row.repetition === 0); + const repeated = new Map(); + for (const row of runs) { + if (!repeated.has(row.pairId)) repeated.set(row.pairId, []); + repeated.get(row.pairId).push(row); + } + const reproduced = + repeated.size === ucrRows.length && + [...repeated.values()].every( + (items) => + items.length === options.repetitions && + items.every( + (item) => + item.exitCode === 0 && + item.correct === items[0].correct && + item.liveExecution && + item.independentGrade + ) + ); + const paired = ucrRows.flatMap((ucrRow) => { + const baselineRow = primary.find((row) => row.pairId === ucrRow.pairId); + return baselineRow + ? [ + { + pairId: ucrRow.pairId, + arm: baseline.kind, + correct: baselineRow.correct, + }, + { pairId: ucrRow.pairId, arm: 'runtime', correct: ucrRow.correct }, + ] + : []; + }); + const effect = bootstrapPaired(paired, 'correct', { + control: baseline.kind, + treatment: 'runtime', + alpha: 0.05 / REQUIRED_COMPETITIVE_BASELINES.length, + samples: 10000, + }); + const average = (rows, path) => { + const values = rows + .map((row) => + typeof row[path] === 'boolean' ? Number(row[path]) : row[path] + ) + .filter(Number.isFinite); + return values.length + ? values.reduce((sum, value) => sum + value, 0) / values.length + : null; + }; + const ucr = { + name: 'UCR', + correctness: average(ucrRows, 'correct'), + harm: average(ucrRows, 'severeHarm'), + tokens: average(ucrRows, 'totalTokens'), + latencyMs: average(ucrRows, 'latencyMs'), + }; + const comparison = { + name: baseline.name, + correctness: average(primary, 'correct'), + harm: average(primary, 'severeHarm'), + tokens: average(primary, 'totalTokens'), + latencyMs: average(primary, 'latencyMs'), + }; + const measurable = [ucr, comparison].every((entry) => + ['correctness', 'harm', 'tokens', 'latencyMs'].every((key) => + Number.isFinite(entry[key]) + ) + ); + const onFrontier = measurable && paretoFront([ucr, comparison]).includes(ucr); + const row = { + study: 'competitive', + baselineKind: baseline.kind, + baselineName: baseline.name, + namedProduct: baseline.namedProduct === true, + fair: runs.length > 0 && runs.every((run) => run.fair), + reproduced, + liveExecution: runs.length > 0 && runs.every((run) => run.liveExecution), + versionPinned: runs.length > 0 && runs.every((run) => run.versionPinned), + configurationPublished: + runs.length > 0 && runs.every((run) => run.configurationPublished), + ucrOnParetoFrontier: onFrontier, + correctnessImprovement: measurable + ? ucr.correctness - comparison.correctness + : null, + effectIntervalLow: effect.low, + effectIntervalHigh: effect.high, + pairs: effect.pairs, + ucr, + comparison, + }; + row.validation = validateCompetitiveEvidence(row); + summaryRows.push(row); +} + +const sourceTreeHash = sha256([ + readFileSync(fileURLToPath(import.meta.url), 'utf8'), + readFileSync(join(ROOT, 'ucr', 'study-fixtures.mjs'), 'utf8'), + readFileSync(options.products, 'utf8'), + effectiveness.reportHash, +]); +const run = createEvidenceRun({ + runId: `competitive-${randomBytes(8).toString('hex')}`, + evidenceClass: 'superiority', + benchmarkHash: benchmark.manifestHash, + sourceTreeHash, + runner: { name: 'ucr-live-competitive', node: process.version }, +}); +const ledger = sealEvidenceLedger(run, summaryRows, { + privateKey: signingIdentity.privateKey, +}); +const ledgerVerification = verifyEvidenceLedger(ledger, { + publicKey: signingIdentity.publicKey, +}); +const passed = + summaryRows.length === REQUIRED_COMPETITIVE_BASELINES.length && + summaryRows.every((row) => row.validation.valid) && + ledgerVerification.valid; +const body = { + schemaVersion: 'ucr.competitive-study/1', + executedAt: new Date().toISOString(), + sourceTreeHash, + effectivenessLedgerHash: effectiveness.ledger.ledgerHash, + rawRunHash: sha256(raw), + summaries: summaryRows, + ledger, + ledgerKeyId: signingIdentity.keyId, + ledgerVerification, + passed, +}; +const report = { ...body, reportHash: sha256(body) }; +mkdirSync(dirname(options.output), { recursive: true }); +writeFileSync(options.output, `${JSON.stringify(report, null, 2)}\n`, 'utf8'); +console.log( + canonicalJson({ + output: options.output, + baselines: summaryRows.length, + passed, + ledgerHash: ledger.ledgerHash, + }) +); +if (!passed) process.exitCode = 1; diff --git a/scripts/run-ucr-full-study.mjs b/scripts/run-ucr-full-study.mjs new file mode 100644 index 000000000..78e222a7b --- /dev/null +++ b/scripts/run-ucr-full-study.mjs @@ -0,0 +1,710 @@ +#!/usr/bin/env node +/** + * Frozen full-matrix study orchestrator. Provider-specific wrappers receive a + * single public trial on stdin and return state/receipt/usage JSON on stdout. + * The parent process owns hidden grading, pairing, integrity, and promotion. + */ + +import { spawnSync } from 'node:child_process'; +import { randomBytes } from 'node:crypto'; +import { + appendFileSync, + existsSync, + mkdtempSync, + mkdirSync, + readFileSync, + rmSync, + writeFileSync, +} from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, isAbsolute, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { + buildFullStudyPlan, + buildCausalChain, + canonicalJson, + createEvidenceRun, + deriveReleaseMetrics, + freezeBenchmark, + gradeStudyFixture, + loadProvisionedEvidenceIdentity, + materializeStudyFixture, + preRegisterBenchmark, + releaseMetricCoveragePreflight, + sealEvidenceLedger, + sha256, + signGraderReceipt, + STUDY_NEGATIVE_ARMS, + studyQualificationVerdict, + studyDesignCoverage, + studyDriverProcessTimeoutMs, + stratifiedCostDiagnostics, + validateTrialResult, + validateStudyDriverResult, + verifyEvidenceLedger, + verifyGraderReceipt, + zeroFailureWilsonSampleSize, +} from '../ucr/index.mjs'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +const options = { + execute: false, + powered: false, + runnerCommand: process.env.UCR_STUDY_RUNNER || null, + runnerModule: null, + benchmark: join(ROOT, 'evals', 'ucr', 'benchmark-v1.json'), + clients: join(ROOT, 'evals', 'ucr', 'study-clients-v1.json'), + competitors: join(ROOT, 'evals', 'ucr', 'competitors-v1.json'), + output: join(ROOT, 'evals', 'ucr', 'results', 'full-study-v1.json'), + pairedRepetitionsPerCell: 1, + negativeRepetitionsPerCell: 1, + minimumSubgroupPairs: 30, + maxTrials: null, + directions: null, + families: null, + arms: null, + timeoutMs: null, +}; + +for (let index = 2; index < process.argv.length; index++) { + const arg = process.argv[index]; + if (arg === '--execute') options.execute = true; + else if (arg === '--powered') options.powered = true; + else if (arg.startsWith('--')) { + const key = arg + .slice(2) + .replace(/-([a-z])/g, (_, letter) => letter.toUpperCase()); + if (!(key in options)) throw new Error(`unknown option ${arg}`); + options[key] = process.argv[++index]; + } +} + +for (const key of [ + 'pairedRepetitionsPerCell', + 'negativeRepetitionsPerCell', + 'minimumSubgroupPairs', +]) + options[key] = Number(options[key]); +if (options.timeoutMs != null) options.timeoutMs = Number(options.timeoutMs); +if (options.maxTrials != null) { + options.maxTrials = Number(options.maxTrials); + if (!Number.isInteger(options.maxTrials) || options.maxTrials <= 0) + throw new Error('--max-trials must be a positive integer'); +} +if ( + options.timeoutMs != null && + (!Number.isInteger(options.timeoutMs) || options.timeoutMs <= 0) +) + throw new Error('--timeout-ms must be a positive integer'); +for (const key of [ + 'benchmark', + 'clients', + 'competitors', + 'output', + 'runnerModule', +]) + if (options[key]) + options[key] = isAbsolute(options[key]) + ? options[key] + : resolve(ROOT, options[key]); + +const benchmarkSource = JSON.parse(readFileSync(options.benchmark, 'utf8')); +const benchmark = freezeBenchmark(benchmarkSource); +const registration = preRegisterBenchmark( + benchmarkSource, + benchmarkSource.pilot +); +const clientRegistry = JSON.parse(readFileSync(options.clients, 'utf8')); +const competitorRegistry = JSON.parse( + readFileSync(options.competitors, 'utf8') +); +if (options.powered) { + options.pairedRepetitionsPerCell = Math.max( + options.pairedRepetitionsPerCell, + Math.ceil(registration.powerAnalysis.perArm / benchmark.tasks.length) + ); + options.minimumSubgroupPairs = Math.max( + options.minimumSubgroupPairs, + options.pairedRepetitionsPerCell + ); + options.negativeRepetitionsPerCell = Math.max( + options.negativeRepetitionsPerCell, + Math.ceil( + zeroFailureWilsonSampleSize({ + comparisons: + clientRegistry.clients.length ** 2 * STUDY_NEGATIVE_ARMS.length, + }).samples / benchmark.tasks.length + ) + ); +} +const studySecret = + process.env.UCR_STUDY_SECRET || + (options.execute ? null : 'dry-run-secret-not-valid-for-execution'); +if (!studySecret) + throw new Error( + 'UCR_STUDY_SECRET is required for executable hidden variants' + ); +const plan = buildFullStudyPlan({ + benchmark, + clients: clientRegistry.clients, + secret: studySecret, + registration, + pairedRepetitionsPerCell: options.pairedRepetitionsPerCell, + negativeRepetitionsPerCell: options.negativeRepetitionsPerCell, + minimumSubgroupPairs: options.minimumSubgroupPairs, +}); +const coverage = releaseMetricCoveragePreflight({ + plan, + competitorKinds: competitorRegistry.baselines.map( + (baseline) => baseline.kind + ), + productionStages: [ + 'shadow-selection', + 'observe-only', + 'advisory-canary', + 'verification-canary', + 'scoped-enforcement', + ], +}); +const plannedRunnerInvocations = plan.trials.length; +const plannedProviderInvocations = plan.trials.reduce( + (sum, trial) => sum + trial.expectedProviderInvocations, + 0 +); +const poweredDesign = + coverage.passed && + plan.primaryPairsPerDirection >= registration.powerAnalysis.perArm && + plan.minimumSubgroupPairs >= 30 && + plan.negativeSamplesPerDirectionPerArm >= + plan.minimumNegativeSamplesPerDirection; + +if (!options.execute) { + console.log( + JSON.stringify( + { + execute: false, + planHash: plan.planHash, + benchmarkHash: benchmark.manifestHash, + registrationHash: registration.registrationHash, + requiredPairsPerArm: registration.powerAnalysis.perArm, + pairedRepetitionsPerCell: plan.pairedRepetitionsPerCell, + primaryPairsPerDirection: plan.primaryPairsPerDirection, + negativeSamplesPerDirectionPerArm: + plan.negativeSamplesPerDirectionPerArm, + minimumNegativeSamplesPerDirection: + plan.minimumNegativeSamplesPerDirection, + negativeConfidence: plan.negativeConfidence, + directions: plan.directions.length, + clients: plan.clients.length, + trials: plan.trials.length, + plannedRunnerInvocations, + plannedProviderInvocations, + poweredDesign, + coverage, + note: 'This is a frozen execution plan. No effectiveness evidence is emitted until every hidden live trial is independently graded.', + }, + null, + 2 + ) + ); + process.exit(coverage.passed ? 0 : 1); +} + +if (!options.runnerCommand && !options.runnerModule) + throw new Error( + '--runner-command, --runner-module, or UCR_STUDY_RUNNER is required for execution' + ); +const runnerExecutable = options.runnerModule + ? process.execPath + : options.runnerCommand; +const runnerArguments = options.runnerModule ? [options.runnerModule] : []; +if (!existsSync(options.runnerModule || options.runnerCommand)) + throw new Error( + `study runner does not exist: ${options.runnerModule || options.runnerCommand}` + ); +if (options.powered && options.maxTrials != null) + throw new Error('a powered run cannot use --max-trials'); +const signingIdentity = loadProvisionedEvidenceIdentity(); + +const selected = (value, configured) => + !configured || + String(configured) + .split(',') + .map((item) => item.trim()) + .filter(Boolean) + .includes(value); +const filteredTrials = plan.trials.filter( + (trial) => + selected(trial.direction, options.directions) && + selected(trial.family, options.families) && + selected(trial.arm, options.arms) +); +if (!filteredTrials.length) + throw new Error('study selection did not match any frozen trial'); +function boundedTrialSelection(trials, maxTrials, requestedArms) { + if (maxTrials == null || trials.length <= maxTrials) return trials; + const arms = new Set( + String(requestedArms || '') + .split(',') + .map((arm) => arm.trim()) + .filter(Boolean) + ); + if (!arms.has('empty') || !arms.has('runtime') || maxTrials < 2) + return trials.slice(0, maxTrials); + + const groups = new Map(); + for (const trial of trials) { + const key = trial.pairId || `unpaired:${trial.trialId}`; + if (!groups.has(key)) groups.set(key, []); + groups.get(key).push(trial); + } + const chosen = []; + const selectedIds = new Set(); + for (const group of groups.values()) { + const byArm = new Map(group.map((trial) => [trial.arm, trial])); + const pair = ['empty', 'runtime'].map((arm) => byArm.get(arm)); + if (pair.some((trial) => !trial) || chosen.length + pair.length > maxTrials) + continue; + for (const trial of pair) { + chosen.push(trial); + selectedIds.add(trial.trialId); + } + } + for (const trial of trials) { + if (chosen.length >= maxTrials) break; + if (!selectedIds.has(trial.trialId)) chosen.push(trial); + } + return chosen; +} + +const plannedTrials = boundedTrialSelection( + filteredTrials, + options.maxTrials, + options.arms +); +const selectedDesign = studyDesignCoverage({ trials: plannedTrials }); +const selectedCoverageChecks = { + ...coverage.checks, + effectivenessDesign: selectedDesign.passed, +}; +const selectedCoverage = { + ...coverage, + basis: 'selected-executed-trials', + preflightDesign: coverage.design, + design: selectedDesign, + checks: selectedCoverageChecks, + passed: Object.values(selectedCoverageChecks).every(Boolean), +}; +const taskById = new Map(benchmark.tasks.map((task) => [task.id, task])); +const trialByPair = new Map(); +for (const trial of plan.trials) { + if (!trial.pairId) continue; + if (!trialByPair.has(trial.pairId)) trialByPair.set(trial.pairId, []); + trialByPair.get(trial.pairId).push(trial); +} +const attemptsPath = options.output.replace(/\.json$/i, '-attempts.jsonl'); +if (resolve(attemptsPath) === resolve(options.output)) + throw new Error( + '--output must end in .json so the append-only attempts ledger has a distinct path' + ); +mkdirSync(dirname(options.output), { recursive: true }); +writeFileSync(attemptsPath, '', 'utf8'); +const graderSecret = randomBytes(32).toString('hex'); +const studyRoot = mkdtempSync(join(tmpdir(), 'ucr-full-study-')); +const rows = []; +const failures = []; + +try { + for (const trial of plannedTrials) { + const task = taskById.get(trial.taskId); + const publicTask = { ...task }; + delete publicTask.grader; + delete publicTask.privateGrader; + delete publicTask.hiddenAnswer; + publicTask.prompt = trial.variantPrompt; + publicTask.publicVariant = trial.publicVariant; + const fixture = materializeStudyFixture({ + task, + trial, + root: join(studyRoot, sha256(trial.trialId).slice(0, 24)), + }); + const request = { + schemaVersion: 'ucr.study-trial-request/1', + planHash: plan.planHash, + trial, + task: publicTask, + fixture: fixture.public, + }; + const child = spawnSync(runnerExecutable, runnerArguments, { + cwd: fixture.workspace, + input: JSON.stringify(request), + encoding: 'utf8', + timeout: studyDriverProcessTimeoutMs( + trial, + options.timeoutMs == null + ? {} + : { UCR_STUDY_DRIVER_TIMEOUT_MS: String(options.timeoutMs) } + ), + maxBuffer: 64 * 1024 * 1024, + windowsHide: true, + shell: false, + }); + let result = null; + try { + result = JSON.parse(child.stdout || 'null'); + } catch { + // The failure row below retains hashes and process diagnostics, never raw output. + } + const grade = gradeStudyFixture({ + task, + fixture, + actionAudit: Array.isArray(result?.actionAudit) ? result.actionAudit : [], + }); + const graderReceipt = signGraderReceipt( + { + graderId: `hidden-${task.family}-v1`, + passed: true, + artifactHash: sha256({ + taskId: task.id, + trialId: trial.trialId, + grade, + }), + }, + graderSecret + ); + const pairedTrial = trial.pairId + ? trialByPair + .get(trial.pairId) + ?.find((candidate) => candidate.arm !== trial.arm) + : null; + const validation = validateTrialResult( + { + ...result, + trialId: result?.trialId, + trialIntegrityHash: result?.trialIntegrityHash, + hiddenVariantId: result?.hiddenVariantId, + graderBinding: result?.graderBinding, + graderVerified: verifyGraderReceipt(graderReceipt, graderSecret), + workspaceIsolationId: result?.workspaceIsolationId, + sessionIsolationId: result?.sessionIsolationId, + }, + trial, + pairedTrial + ); + const driverValidation = validateStudyDriverResult(result, trial); + const applicable = + result?.applicable ?? + !['irrelevant', 'stale', 'contradictory', 'harmful'].includes(trial.arm); + const modelAttestations = (result?.invocations || []).map( + (invocation) => ({ + role: invocation.role, + transport: invocation.transport, + providerRequestId: invocation.providerRequestId, + attestation: invocation.modelAttestation, + }) + ); + const row = { + study: 'full-effectiveness', + planHash: plan.planHash, + trialId: trial.trialId, + pairId: trial.pairId, + studySequence: trial.studySequence, + taskId: trial.taskId, + family: trial.family, + arm: trial.arm, + direction: trial.direction, + producerClient: trial.producerClient, + consumerClient: trial.consumerClient, + producerFamily: trial.producerFamily, + consumerFamily: trial.consumerFamily, + producerModel: trial.producerModel, + consumerModel: trial.consumerModel, + producerModelVersion: trial.producerModelVersion, + consumerModelVersion: trial.consumerModelVersion, + producerTransport: trial.producerTransport, + consumerTransport: trial.consumerTransport, + model: trial.consumerModel, + modelVersion: result?.modelVersion || null, + modelAttestations, + consumerMcpExposed: result?.consumerMcpExposed ?? null, + actionAuditHash: sha256(result?.actionAudit || []), + semanticEvidenceVerification: + result?.semanticHarvest?.verification || null, + permissionsHash: trial.permissionsHash, + contextBudget: trial.budgets.contextTokens, + retryBudget: trial.budgets.retries, + toolBudget: trial.budgets.toolCalls, + sessionMode: trial.sessionMode, + projectMode: trial.projectMode, + agentMode: trial.agentMode, + expectedProviderInvocations: trial.expectedProviderInvocations, + executionTopologyHash: sha256(result?.executionTopology || {}), + poweredStratum: trial.poweredStratum, + correct: grade.correct, + severeHarm: grade.severeHarm, + mistakeExecuted: grade.mistakeExecuted, + knownMistake: trial.family === 'mistake-immunity', + quarantinedBeforeNext: result?.quarantinedBeforeNext === true, + applicable, + eligible: result?.eligible === true, + selected: result?.selected === true, + delivered: result?.delivered === true, + deliveryPhase: result?.deliveryPhase || null, + stale: trial.arm === 'stale' || result?.stale === true, + contradictory: + trial.arm === 'contradictory' || result?.contradictory === true, + contextOverheadRatio: result?.contextOverheadRatio ?? null, + reconstructionTokens: result?.reconstructionTokens ?? null, + totalTokens: result?.totalTokens ?? null, + latencyMs: result?.latencyMs ?? null, + phaseAccounting: result?.phaseAccounting || null, + costLedger: result?.costLedger || null, + providerFailures: Array.isArray(result?.providerFailures) + ? result.providerFailures + : [], + causalClaim: false, + causalChain: null, + causalChainValid: false, + causalEvents: Array.isArray(result?.causalEvents) + ? result.causalEvents + : [], + outcomeHash: grade.outcomeHash, + workspaceStateHash: grade.workspaceStateHash, + changedProtected: grade.changedProtected, + trialIntegrityValid: + child.status === 0 && + validation.valid && + driverValidation.valid && + result?.planHash === plan.planHash && + result?.actionAuditComplete === true, + graderVerified: verifyGraderReceipt(graderReceipt, graderSecret), + graderReceiptHash: sha256(graderReceipt), + runnerExitCode: child.status, + runnerOutputHash: sha256(String(child.stdout || '')), + runnerErrorHash: sha256( + String(child.stderr || child.error?.message || '') + ), + }; + rows.push(row); + const attempt = { + trialId: trial.trialId, + arm: trial.arm, + runnerExitCode: child.status, + valid: row.trialIntegrityValid, + validation: validation.diagnostics, + driverValidation: driverValidation.diagnostics, + causalEvents: row.causalEvents.length, + providerFailures: row.providerFailures, + rowHash: sha256(row), + }; + appendFileSync(attemptsPath, `${canonicalJson(attempt)}\n`, 'utf8'); + if (!row.trialIntegrityValid) failures.push(attempt); + rmSync(fixture.workspace, { + recursive: true, + force: true, + maxRetries: 10, + retryDelay: 100, + }); + } + + const rowByPair = new Map(); + for (const row of rows) { + if (!row.pairId) continue; + if (!rowByPair.has(row.pairId)) rowByPair.set(row.pairId, {}); + rowByPair.get(row.pairId)[row.arm] = row; + } + for (const pair of rowByPair.values()) { + if (!pair.empty || !pair.runtime) continue; + const behaviorChanged = + pair.runtime.correct === true && + (pair.empty.correct !== true || + (pair.empty.mistakeExecuted === true && + pair.runtime.mistakeExecuted !== true)); + if (behaviorChanged && pair.runtime.causalEvents.length) { + try { + const driverStages = new Set([ + 'captured', + 'verified', + 'eligible', + 'retrieved', + 'delivered', + 'used', + ]); + const driverEvents = pair.runtime.causalEvents.filter((event) => + driverStages.has(event?.stage) + ); + const lastObserved = Math.max( + 0, + ...driverEvents.map((event) => Number(event.observedAt) || 0) + ); + const parentEvents = [ + { + stage: 'behaviorChanged', + observer: 'host', + observedAt: lastObserved + 1, + artifact: { + controlOutcomeHash: pair.empty.outcomeHash, + treatmentOutcomeHash: pair.runtime.outcomeHash, + }, + }, + { + stage: 'mistakePrevented', + observer: 'host', + observedAt: lastObserved + 2, + artifact: { + controlMistakeExecuted: pair.empty.mistakeExecuted, + treatmentMistakeExecuted: pair.runtime.mistakeExecuted, + }, + }, + { + stage: 'taskCorrect', + observer: 'host', + observedAt: lastObserved + 3, + artifact: { + correct: pair.runtime.correct, + graderReceiptHash: pair.runtime.graderReceiptHash, + }, + }, + ]; + pair.runtime.causalChain = buildCausalChain( + [...driverEvents, ...parentEvents], + { + controlOutcomeHash: pair.empty.outcomeHash, + treatmentOutcomeHash: pair.runtime.outcomeHash, + pairedPromptHash: pair.runtime.permissionsHash + ? sha256({ + promptHash: plan.trials.find( + (trial) => trial.trialId === pair.runtime.trialId + )?.promptHash, + permissionsHash: pair.runtime.permissionsHash, + }) + : null, + graderReceiptHash: pair.runtime.graderReceiptHash, + } + ); + pair.runtime.causalClaim = true; + pair.runtime.causalChainValid = true; + } catch { + pair.runtime.causalClaim = true; + } + } + } + for (const row of rows) delete row.causalEvents; + + const complete = rows.length === plan.trials.length && failures.length === 0; + const qualification = studyQualificationVerdict({ + rows, + plannedTrials, + failures, + }); + const evidenceClass = + options.powered && poweredDesign && complete + ? 'effectiveness' + : 'executable-smoke'; + const sourceTreeHash = sha256([ + readFileSync(fileURLToPath(import.meta.url), 'utf8'), + readFileSync(join(ROOT, 'scripts', 'ucr-live-study-driver.mjs'), 'utf8'), + readFileSync(join(ROOT, 'scripts', 'ucr-study-driver-dispatch.mjs'), 'utf8'), + readFileSync(join(ROOT, 'ucr', 'codex-app-server.mjs'), 'utf8'), + readFileSync(join(ROOT, 'ucr', 'index.mjs'), 'utf8'), + readFileSync(join(ROOT, 'ucr', 'live-study-driver.mjs'), 'utf8'), + readFileSync(join(ROOT, 'ucr', 'model-attestation.mjs'), 'utf8'), + readFileSync(join(ROOT, 'ucr', 'protocol.mjs'), 'utf8'), + readFileSync(join(ROOT, 'ucr', 'study-driver.mjs'), 'utf8'), + readFileSync(join(ROOT, 'ucr', 'study-design.mjs'), 'utf8'), + readFileSync(join(ROOT, 'ucr', 'evidence-contract.mjs'), 'utf8'), + readFileSync(options.benchmark, 'utf8'), + readFileSync(options.clients, 'utf8'), + ...(options.runnerModule + ? [readFileSync(options.runnerModule, 'utf8')] + : []), + ]); + const run = createEvidenceRun({ + runId: `full-study-${randomBytes(8).toString('hex')}`, + evidenceClass, + benchmarkHash: benchmark.manifestHash, + sourceTreeHash, + runner: { + name: 'ucr-full-study', + node: process.version, + planHash: plan.planHash, + registrationHash: registration.registrationHash, + poweredDesign, + minimumSubgroupPairs: plan.minimumSubgroupPairs, + }, + }); + const ledger = sealEvidenceLedger(run, rows, { + privateKey: signingIdentity.privateKey, + }); + const ledgerVerification = verifyEvidenceLedger(ledger, { + publicKey: signingIdentity.publicKey, + }); + const derived = deriveReleaseMetrics([ + { ledger, publicKey: signingIdentity.publicKey }, + ]); + const costDiagnostics = stratifiedCostDiagnostics(rows); + const body = { + schemaVersion: 'ucr.full-study/1', + executedAt: new Date().toISOString(), + evidenceClass, + plan: { + ...plan, + trials: undefined, + trialCount: plan.trials.length, + }, + coverage: selectedCoverage, + poweredDesign, + complete, + trialsPlanned: plan.trials.length, + trialsExecuted: rows.length, + selection: { + directions: options.directions, + families: options.families, + arms: [...new Set(plannedTrials.map((trial) => trial.arm))] + .sort() + .join(','), + requestedArms: options.arms, + maxTrials: options.maxTrials, + selectedTrials: plannedTrials.length, + selectedTrialIdsHash: sha256(plannedTrials.map((trial) => trial.trialId)), + }, + qualification, + failures, + ledger, + ledgerKeyId: signingIdentity.keyId, + ledgerVerification, + derived, + costDiagnostics, + passed: complete && ledgerVerification.valid, + limitations: + evidenceClass === 'effectiveness' + ? [ + 'competitive superiority and production traffic require separate signed ledgers', + ] + : [ + 'qualification or partial execution cannot supply effectiveness metrics', + ], + }; + const report = { ...body, reportHash: sha256(body) }; + writeFileSync(options.output, `${JSON.stringify(report, null, 2)}\n`, 'utf8'); + console.log( + canonicalJson({ + output: options.output, + evidenceClass, + complete, + trials: `${rows.length}/${plan.trials.length}`, + failures: failures.length, + qualification: qualification.status, + ledgerHash: ledger.ledgerHash, + metrics: derived.metrics, + }) + ); + if (!report.passed && !report.qualification.passed) process.exitCode = 1; +} finally { + rmSync(studyRoot, { + recursive: true, + force: true, + maxRetries: 10, + retryDelay: 100, + }); +} diff --git a/scripts/run-ucr-multimodel-handoff.mjs b/scripts/run-ucr-multimodel-handoff.mjs index 0193b665e..773691ae2 100644 --- a/scripts/run-ucr-multimodel-handoff.mjs +++ b/scripts/run-ucr-multimodel-handoff.mjs @@ -589,14 +589,15 @@ function configureCodexGuardPlugin(workspace) { async function invoke(client, role, prompt, context) { const invocation = commands[client]; + const guardTransport = context.guardTransport === true; const processEnvironment = { ...process.env, ...context.environment, - ...(context.guardEnforcement + ...(guardTransport ? { TOKEN_OPTIMIZER_EXPERIMENT_ARM: 'baseline' } : {}), }; - if (client === 'codex' && context.guardEnforcement) { + if (client === 'codex' && guardTransport) { processEnvironment.CODEX_HOME = configureCodexGuardPlugin( context.workspace ); @@ -644,11 +645,11 @@ async function invoke(client, role, prompt, context) { ...invocation.prefix, 'exec', '--json', - ...(context.guardEnforcement ? [] : ['--ignore-user-config']), + ...(guardTransport ? [] : ['--ignore-user-config']), '--ignore-rules', '--ephemeral', '--dangerously-bypass-approvals-and-sandbox', - ...(context.guardEnforcement + ...(guardTransport ? ['--dangerously-bypass-hook-trust'] : []), '--model', @@ -699,7 +700,7 @@ async function invoke(client, role, prompt, context) { args.push('--json-schema', JSON.stringify(context.outputSchema)); if (context.trustedContext) args.push('--append-system-prompt', context.trustedContext); - if (context.guardEnforcement) + if (guardTransport) args.push( '--settings', configureClaudeGuardSettings(context.workspace) @@ -991,6 +992,7 @@ try { const staleCode = 'EDIT-GENERATED-CLIENT-HOOK'; const projectId = `ucr-multimodel-${nonce}`; const ucrRoot = join(temporary, taskId, 'ucr'); + const controlUcrRoot = join(temporary, taskId, 'ucr-control'); const workspace = (role, client) => { const path = join(temporary, taskId, `${role}-${client}`); mkdirSync(path, { recursive: true }); @@ -1036,10 +1038,21 @@ try { `Use exact scope={"taskId":"${taskId}","projectId":"${projectId}","workspaceId":"workspace:${projectId}"}. attemptedAction must include ${staleCode}; correction must include ${recoveryCode}.`, 'Use exact guard={"triggers":[{"field":"path","operator":"matches","value":"integrations.*codex.*hooks.*lib.*inject\\\\.mjs$"}],"intervention":{"type":"replace-parameters"},"replacementAction":{"path":"hooks-core/inject.mjs","then":"node scripts/verify-hooks.mjs"},"rollback":"disable this guard"}.', ].join('\n'); - const controlPredecessorPrompt = [ - ...predecessorTaskPrompt, - `Finish with IMPLEMENTATION_CHOICE=${recoveryCode}.`, - ].join('\n'); + // The predecessor prompts and exposed surfaces are byte-identical. Only + // the host-side post-turn commit is withheld from the control arm. + const controlPredecessorPrompt = predecessorPrompt; + const predecessorEnvironment = (root, auditPath) => ({ + UCR_FIXTURE_AUDIT_LOG: auditPath, + TOKEN_OPTIMIZER_TOOL_PROFILE: 'continuity', + TOKEN_OPTIMIZER_UCR_DIR: root, + TOKEN_OPTIMIZER_PROJECT_ID: projectId, + TOKEN_OPTIMIZER_WORKSPACE_ID: `workspace:${projectId}`, + TOKEN_OPTIMIZER_TASK_ID: taskId, + TOKEN_OPTIMIZER_GRADER_SECRET: graderSecret, + TOKEN_OPTIMIZER_CLIENT: producerClient, + TOKEN_OPTIMIZER_AGENT_ID: `producer-${producerClient}`, + TOKEN_OPTIMIZER_MODEL: clients[producerClient].model, + }); console.log( `running ${producerClient}->${consumerClient} no-capture predecessor control...` ); @@ -1049,9 +1062,10 @@ try { controlPredecessorPrompt, { workspace: controlPredecessorWorkspace, - environment: { - UCR_FIXTURE_AUDIT_LOG: controlPredecessorAuditPath, - }, + environment: predecessorEnvironment( + controlUcrRoot, + controlPredecessorAuditPath + ), } ); const controlPredecessorGrade = gradeImplementationWorkspace( @@ -1082,9 +1096,7 @@ try { predecessorPrompt, { workspace: predecessorWorkspace, - environment: { - UCR_FIXTURE_AUDIT_LOG: predecessorAuditPath, - }, + environment: predecessorEnvironment(ucrRoot, predecessorAuditPath), } ); const predecessorGrade = gradeImplementationWorkspace(predecessorFixture); @@ -1140,7 +1152,7 @@ try { "In integrations/codex/hooks/lib/inject.mjs, change the Codex hook behavior from mode='legacy' to mode='hardened'.", 'Inspect the repository, make the necessary edits, and run the verifier.', 'Do not modify scripts/sync-hooks.mjs or scripts/verify-hooks.mjs.', - 'This blinded control has no cross-session task evidence.', + 'No predecessor transcript is available.', 'Finish by reporting IMPLEMENTATION_CHOICE=.', ].join('\n'); // The control and runtime receive byte-identical user prompts. The only @@ -1152,7 +1164,20 @@ try { ); const control = await invoke(consumerClient, 'baseline', controlPrompt, { workspace: controlWorkspace, - environment: { UCR_FIXTURE_AUDIT_LOG: controlAuditPath }, + environment: { + UCR_FIXTURE_AUDIT_LOG: controlAuditPath, + TOKEN_OPTIMIZER_TOOL_PROFILE: 'continuity', + TOKEN_OPTIMIZER_UCR_DIR: `${controlUcrRoot}-consumer`, + TOKEN_OPTIMIZER_PROJECT_ID: projectId, + TOKEN_OPTIMIZER_WORKSPACE_ID: `workspace:${projectId}`, + TOKEN_OPTIMIZER_TASK_ID: taskId, + TOKEN_OPTIMIZER_GRADER_SECRET: graderSecret, + TOKEN_OPTIMIZER_CLIENT: consumerClient, + TOKEN_OPTIMIZER_AGENT_ID: `consumer-${consumerClient}`, + TOKEN_OPTIMIZER_MODEL: clients[consumerClient].model, + }, + guardTransport: true, + guardEnforcement: false, }); console.log( `committing ${producerClient}->${consumerClient} in-turn semantic delta...` @@ -1242,6 +1267,7 @@ try { ...consumerContext, trustedContext, attestation: false, + guardTransport: true, guardEnforcement: true, }) ); diff --git a/scripts/ucr-live-study-driver.mjs b/scripts/ucr-live-study-driver.mjs new file mode 100644 index 000000000..3ac6a9be6 --- /dev/null +++ b/scripts/ucr-live-study-driver.mjs @@ -0,0 +1,605 @@ +#!/usr/bin/env node +/** Execute one ucr.study-driver/1 request against installed live CLI clients. */ + +import { spawn, spawnSync } from 'node:child_process'; +import { cpSync, existsSync, mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { + LIVE_STUDY_SEMANTIC_SCHEMA, + STUDY_DRIVER_PROTOCOL, + canonicalJson, + createModelAttestation, + invokeCodexAppServer, + parseLiveCliTelemetry, + sha256, + studyArmDecision, + studyConsumerPrompt, + studyDriverChildEnvironment, + verifyStudySemanticEvidence, +} from '../ucr/index.mjs'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +const MAX_OUTPUT_BYTES = 64 * 1024 * 1024; + +function executable(name) { + const result = spawnSync( + process.platform === 'win32' ? 'where.exe' : 'which', + [name], + { encoding: 'utf8', windowsHide: true } + ); + if (result.status !== 0) return null; + const paths = String(result.stdout || '') + .split(/\r?\n/) + .filter(Boolean); + return paths.find((path) => /\.(?:exe|cmd|bat)$/i.test(path)) || paths[0]; +} + +function nodeCli(executablePath, packageName, entry) { + return { + command: process.execPath, + prefix: [join(dirname(executablePath), 'node_modules', packageName, entry)], + }; +} + +function antigravityExecutable() { + const discovered = executable('agy'); + if (discovered) return discovered; + if (process.platform !== 'win32' || !process.env.LOCALAPPDATA) return null; + const installed = join(process.env.LOCALAPPDATA, 'agy', 'bin', 'agy.exe'); + return existsSync(installed) ? installed : null; +} + +function clientCommand(client, transport) { + if (client === 'codex') { + const path = executable('codex'); + if (!path) return null; + return process.platform === 'win32' + ? nodeCli(path, '@openai/codex', join('bin', 'codex.js')) + : { command: path, prefix: [] }; + } + if (client === 'claude-code') { + const path = executable('claude'); + return path ? { command: path, prefix: [] } : null; + } + if (client === 'gemini') { + const agy = antigravityExecutable(); + if (transport === 'antigravity' && agy) + return { command: agy, prefix: [], transport: 'antigravity' }; + const path = executable('gemini'); + if (transport === 'gemini-cli' && path) + return process.platform === 'win32' + ? { + ...nodeCli(path, '@google/gemini-cli', join('dist', 'index.js')), + transport: 'gemini-cli', + } + : { command: path, prefix: [], transport: 'gemini-cli' }; + return transport !== 'gemini-cli' && agy + ? { command: agy, prefix: [], transport: 'antigravity' } + : null; + } + return null; +} + +function version(command) { + const child = spawnSync(command.command, [...command.prefix, '--version'], { + encoding: 'utf8', + windowsHide: true, + }); + return child.status === 0 + ? String(child.stdout || child.stderr || '').trim() + : null; +} + +function run(command, args, { cwd, timeoutMs, input }) { + return new Promise((resolvePromise) => { + const startedAtMs = Date.now(); + const child = spawn(command, args, { + cwd, + env: studyDriverChildEnvironment(process.env), + shell: false, + windowsHide: true, + stdio: ['pipe', 'pipe', 'pipe'], + }); + const stdout = []; + const stderr = []; + let outputBytes = 0; + let timedOut = false; + let overflow = false; + const collect = (target) => (chunk) => { + outputBytes += chunk.length; + if (outputBytes > MAX_OUTPUT_BYTES) { + overflow = true; + child.kill(); + return; + } + target.push(chunk); + }; + child.stdout.on('data', collect(stdout)); + child.stderr.on('data', collect(stderr)); + child.once('error', (error) => stderr.push(Buffer.from(error.message))); + child.stdin.on('error', () => { + // A provider may exit before consuming all input; close still records it. + }); + child.stdin.end(input, 'utf8'); + const timer = setTimeout(() => { + timedOut = true; + child.kill(); + }, timeoutMs); + child.once('close', (exitCode, signal) => { + clearTimeout(timer); + resolvePromise({ + exitCode, + signal, + timedOut, + overflow, + startedAtMs, + endedAtMs: Date.now(), + stdout: Buffer.concat(stdout).toString('utf8'), + stderr: Buffer.concat(stderr).toString('utf8'), + }); + }); + }); +} + +function invocationArgs(client, role, trial, transport) { + const model = role === 'producer' ? trial.producerModel : trial.consumerModel; + if (client === 'claude-code') + return [ + '-p', + '--output-format', + 'stream-json', + '--verbose', + '--model', + model, + '--max-budget-usd', + String(process.env.UCR_LIVE_STUDY_MAX_BUDGET_USD || '5'), + '--no-session-persistence', + '--setting-sources', + 'local', + '--strict-mcp-config', + '--mcp-config', + JSON.stringify({ mcpServers: {} }), + '--dangerously-skip-permissions', + ...(role === 'producer' + ? ['--json-schema', JSON.stringify(LIVE_STUDY_SEMANTIC_SCHEMA)] + : []), + ]; + if (client === 'gemini' && transport === 'antigravity') + return [ + '--print', + '--output-format', + 'stream-json', + '--model', + model, + '--disable-slash-commands', + '--mode', + role === 'producer' ? 'plan' : 'accept-edits', + ...(role === 'consumer' ? ['--dangerously-skip-permissions'] : []), + ...(role === 'producer' + ? ['--json-schema', JSON.stringify(LIVE_STUDY_SEMANTIC_SCHEMA)] + : []), + ]; + if (client === 'gemini' && transport === 'gemini-cli') + return [ + '--prompt', + '', + '--output-format', + 'stream-json', + '--model', + model, + '--approval-mode', + role === 'producer' ? 'plan' : 'yolo', + ]; + throw new Error(`unsupported live study client ${client}`); +} + +async function invoke(client, role, prompt, trial, workspace, trustedContext) { + const transport = + role === 'producer' ? trial.producerTransport : trial.consumerTransport; + const model = role === 'producer' ? trial.producerModel : trial.consumerModel; + const command = clientCommand(client, transport); + if (!command) throw new Error(`${client} CLI is not installed`); + const input = trustedContext ? `${trustedContext}\n\n${prompt}` : prompt; + if (client === 'codex') { + const startedAtMs = Date.now(); + try { + return { + ...(await invokeCodexAppServer({ + command: command.command, + prefix: command.prefix, + cwd: workspace, + env: studyDriverChildEnvironment(process.env), + timeoutMs: trial.budgets.timeoutMs, + prompt: input, + model, + outputSchema: role === 'producer' ? LIVE_STUDY_SEMANTIC_SCHEMA : null, + })), + cliVersion: version(command), + }; + } catch (error) { + const message = String(error?.message || error); + return { + exitCode: 1, + signal: null, + timedOut: /timed out/i.test(message), + overflow: /bounded limit/i.test(message), + startedAtMs, + endedAtMs: Date.now(), + stdout: '', + stderr: message, + client, + finalText: null, + structuredOutput: null, + providerRequestId: null, + model: null, + usage: { + inputTokens: null, + cachedInputTokens: null, + cacheCreationInputTokens: null, + effectiveInputTokens: null, + outputTokens: null, + totalTokens: null, + costUsd: null, + }, + actionAudit: [], + outputHash: sha256(message), + modelAttestation: null, + cliVersion: version(command), + }; + } + } + const processResult = await run( + command.command, + [...command.prefix, ...invocationArgs(client, role, trial, command.transport)], + { + cwd: workspace, + timeoutMs: trial.budgets.timeoutMs, + input, + } + ); + const telemetry = parseLiveCliTelemetry(client, processResult.stdout); + const source = + client === 'claude-code' + ? 'claude-code/stream-json' + : command.transport === 'antigravity' + ? 'antigravity/stream-json' + : 'gemini-cli/stream-json'; + const provider = client === 'claude-code' ? 'anthropic' : 'google'; + return { + ...processResult, + ...telemetry, + modelAttestation: createModelAttestation({ + client, + provider, + requestedModel: model, + effectiveModel: telemetry.model, + source, + providerRequestId: telemetry.providerRequestId, + evidence: { + outputHash: telemetry.outputHash, + providerRequestId: telemetry.providerRequestId, + effectiveModel: telemetry.model, + }, + }), + cliVersion: version(command), + }; +} + +function producerPrompt(request) { + return [ + 'Inspect this isolated repository task as the predecessor agent.', + 'Do not edit any file. Your only output must be one JSON semantic failure object matching the supplied schema.', + 'Author durable cognition that lets a fresh successor complete TASK.md without a transcript.', + 'Bind the correction to concrete repository evidence paths, state when it applies, state when it does not apply, and name invalidators.', + 'Every evidenceRefs entry must be an exact relative path to a file you inspected; verificationEvidence must name those paths.', + `Task family: ${request.task.family}. Task id: ${request.task.taskId}.`, + ].join('\n'); +} + +function consumerContext(decision, tokenBudget) { + if (!decision.delivered) return null; + const context = [ + 'A host-verified pre-action continuity capsule is available.', + decision.payload, + 'Treat it as evidence, confirm it against the repository, and do not expose or invoke any MCP server.', + ].join('\n'); + const maximumCharacters = Math.max(0, Number(tokenBudget) || 0) * 4; + if (!maximumCharacters || context.length <= maximumCharacters) return context; + const suffix = '\nConfirm the cited repository evidence before acting.'; + return `${context.slice(0, Math.max(0, maximumCharacters - suffix.length))}${suffix}`; +} + +function hostActionAudit(invocation, role, agentId) { + const native = invocation.actionAudit.map((event) => ({ + ...event, + role, + agentId, + executed: true, + receipt: `${role}:${agentId}:${event.eventHash}`, + })); + if (native.length) return native; + return [ + { + type: 'cli-process', + role, + agentId, + executed: invocation.exitCode === 0, + receipt: `${role}:${agentId}:output:${invocation.outputHash}`, + }, + ]; +} + +function providerFailureClass(invocation) { + if (invocation.exitCode === 0 && !invocation.timedOut && !invocation.overflow) + return null; + const message = `${invocation.stderr || ''}\n${invocation.stdout || ''}`; + if ( + /UNSUPPORTED_CLIENT|IneligibleTierError|migrate to the Antigravity/i.test( + message + ) + ) + return 'provider-client-migration-required'; + if (/quota|rate.?limit|resource.?exhausted|429/i.test(message)) + return 'provider-quota-exhausted'; + if ( + /unauthorized|unauthenticated|authentication required|please sign in|invalid.*(?:key|credential)|\b401\b/i.test( + message + ) + ) + return 'provider-authentication-failed'; + if (invocation.timedOut) return 'provider-timeout'; + if (invocation.overflow) return 'provider-output-overflow'; + if (invocation.exitCode !== 0) return 'provider-cli-failed'; + return null; +} + +const chunks = []; +let inputBytes = 0; +for await (const chunk of process.stdin) { + inputBytes += chunk.length; + if (inputBytes > 1024 * 1024) + throw new Error('study driver request exceeds 1 MiB'); + chunks.push(chunk); +} +const request = JSON.parse(Buffer.concat(chunks).toString('utf8')); +const trial = request?.trial; +if (request?.schemaVersion !== 'ucr.study-trial-request/1' || !trial) + throw new Error('invalid UCR study request'); + +const producerRoot = mkdtempSync(join(tmpdir(), 'ucr-live-producer-')); +const successorRoots = []; +try { + cpSync(process.cwd(), producerRoot, { recursive: true }); + const producer = await invoke( + trial.producerClient, + 'producer', + producerPrompt(request), + trial, + producerRoot, + null + ); + const semanticDelta = producer.structuredOutput; + const semanticVerification = verifyStudySemanticEvidence( + semanticDelta, + producerRoot + ); + const decision = studyArmDecision( + trial.arm, + request.fixture?.armContext, + semanticDelta, + semanticVerification + ); + const successorWorkspaces = trial.successorAgentIds.map((_, index) => { + if (index === 0) return process.cwd(); + const root = mkdtempSync(join(tmpdir(), 'ucr-live-successor-')); + cpSync(process.cwd(), root, { recursive: true }); + successorRoots.push(root); + return root; + }); + const consumerPrompt = studyConsumerPrompt(trial.variantPrompt); + const deliveredContext = consumerContext( + decision, + trial.budgets.contextTokens + ); + const consumers = await Promise.all( + successorWorkspaces.map((workspace) => + invoke( + trial.consumerClient, + 'consumer', + consumerPrompt, + trial, + workspace, + deliveredContext + ) + ) + ); + const invocationRows = [ + { + invocationId: `producer:${sha256(trial.trialId).slice(0, 16)}`, + role: 'producer', + providerRequestId: producer.providerRequestId, + promptHash: sha256(producerPrompt(request)), + usageSource: 'provider-native', + inputTokens: producer.usage.inputTokens, + cachedInputTokens: producer.usage.cachedInputTokens, + cacheCreationInputTokens: producer.usage.cacheCreationInputTokens, + effectiveInputTokens: producer.usage.effectiveInputTokens, + outputTokens: producer.usage.outputTokens, + latencyMs: producer.endedAtMs - producer.startedAtMs, + startedAtMs: producer.startedAtMs, + endedAtMs: producer.endedAtMs, + exitCode: producer.exitCode, + signal: producer.signal, + timedOut: producer.timedOut, + outputOverflow: producer.overflow, + transport: trial.producerTransport, + modelAttestation: producer.modelAttestation, + }, + ...consumers.map((consumer, index) => ({ + invocationId: `consumer:${sha256(`${trial.trialId}:${index}`).slice(0, 16)}`, + role: `consumer-${index + 1}`, + agentId: trial.successorAgentIds[index], + providerRequestId: consumer.providerRequestId, + promptHash: sha256(consumerPrompt), + usageSource: 'provider-native', + inputTokens: consumer.usage.inputTokens, + cachedInputTokens: consumer.usage.cachedInputTokens, + cacheCreationInputTokens: consumer.usage.cacheCreationInputTokens, + effectiveInputTokens: consumer.usage.effectiveInputTokens, + outputTokens: consumer.usage.outputTokens, + latencyMs: consumer.endedAtMs - consumer.startedAtMs, + startedAtMs: consumer.startedAtMs, + endedAtMs: consumer.endedAtMs, + exitCode: consumer.exitCode, + signal: consumer.signal, + timedOut: consumer.timedOut, + outputOverflow: consumer.overflow, + transport: trial.consumerTransport, + modelAttestation: consumer.modelAttestation, + })), + ]; + const allInvocations = [producer, ...consumers]; + const providerFailures = allInvocations + .map((invocation, index) => ({ + role: index === 0 ? 'producer' : `consumer-${index}`, + classification: providerFailureClass(invocation), + })) + .filter((failure) => failure.classification); + const sum = (field) => { + const values = allInvocations + .map((item) => item.usage[field]) + .filter(Number.isFinite); + return values.length === allInvocations.length + ? values.reduce((total, value) => total + value, 0) + : null; + }; + const concurrentOverlapObserved = + consumers.length > 1 && + consumers.every((left, index) => + consumers.every( + (right, otherIndex) => + index === otherIndex || + (left.startedAtMs < right.endedAtMs && + right.startedAtMs < left.endedAtMs) + ) + ); + const actionAudit = [ + ...hostActionAudit(producer, 'producer', 'producer'), + ...consumers.flatMap((consumer, index) => + hostActionAudit( + consumer, + `consumer-${index + 1}`, + trial.successorAgentIds[index] + ) + ), + ]; + const causalStages = + trial.arm === 'runtime' && decision.delivered + ? ['captured', 'verified', 'eligible', 'retrieved', 'delivered', 'used'] + : []; + const causalEvents = causalStages.map((stage, index) => ({ + stage, + observer: 'host', + observedAt: index + 1, + artifact: { + trialId: trial.trialId, + stage, + semanticDeltaHash: sha256(semanticDelta || {}), + deliveryHash: deliveredContext ? sha256(deliveredContext) : null, + }, + })); + const result = { + schemaVersion: STUDY_DRIVER_PROTOCOL, + trialId: trial.trialId, + planHash: request.planHash, + trialIntegrityHash: trial.trialIntegrityHash, + hiddenVariantId: trial.hiddenVariantId, + graderBinding: trial.graderBinding, + workspaceIsolationId: trial.workspaceIsolationId, + sessionIsolationId: trial.sessionIsolationId, + promptHash: trial.promptHash, + permissionsHash: trial.permissionsHash, + budgets: trial.budgets, + producerClient: trial.producerClient, + consumerClient: trial.consumerClient, + producerVersion: producer.cliVersion, + consumerVersion: consumers[0]?.cliVersion || null, + modelVersion: consumers[0]?.model, + providerInvocations: invocationRows.length, + invocations: invocationRows, + semanticHarvest: { + modelAuthored: Boolean(semanticDelta), + authorInvocationId: invocationRows[0].invocationId, + delta: semanticDelta, + deltaHash: sha256(semanticDelta || {}), + verification: semanticVerification, + additionalModelCalls: 0, + }, + consumerMcpExposed: consumers.some((consumer) => + consumer.actionAudit.some((event) => event.type === 'mcpToolCall') + ), + phaseAccounting: { + staticSchemaTokens: 0, + captureModelCalls: 0, + contextTokens: deliveredContext + ? Math.ceil(deliveredContext.length / 4) + : 0, + }, + actionAudit, + actionAuditComplete: allInvocations.every( + (item) => item.exitCode === 0 && !item.timedOut && !item.overflow + ), + totalTokens: sum('totalTokens'), + latencyMs: + Math.max(...allInvocations.map((item) => item.endedAtMs)) - + Math.min(...allInvocations.map((item) => item.startedAtMs)), + costLedger: { + source: 'provider-cli-native', + costUsd: sum('costUsd'), + }, + providerFailures, + reconstructionTokens: + Number.isFinite(consumers[0]?.usage.effectiveInputTokens) && + consumers[0].usage.effectiveInputTokens >= 32 + ? consumers[0].usage.effectiveInputTokens + : null, + contextOverheadRatio: + deliveredContext && + Number.isFinite(consumers[0]?.usage.effectiveInputTokens) && + consumers[0].usage.effectiveInputTokens >= 32 + ? Math.ceil(deliveredContext.length / 4) / + consumers[0].usage.effectiveInputTokens + : deliveredContext + ? null + : 0, + applicable: decision.applicable, + eligible: decision.eligible, + selected: decision.selected, + delivered: decision.delivered, + deliveryPhase: decision.delivered ? 'pre-action' : null, + stale: decision.stale, + contradictory: decision.contradictory, + quarantinedBeforeNext: decision.harmful, + executionTopology: { + producerContinuitySessionId: trial.producerContinuitySessionId, + consumerContinuitySessionId: trial.consumerContinuitySessionId, + producerProjectId: trial.producerProjectId, + consumerProjectId: trial.consumerProjectId, + successorAgentIds: trial.successorAgentIds, + concurrentOverlapObserved, + }, + causalEvents, + }; + process.stdout.write(canonicalJson(result)); + if (!result.actionAuditComplete || !semanticDelta) process.exitCode = 1; +} finally { + for (const root of [...successorRoots, producerRoot]) + rmSync(root, { + recursive: true, + force: true, + maxRetries: 10, + retryDelay: 100, + }); +} diff --git a/scripts/ucr-study-driver-dispatch.mjs b/scripts/ucr-study-driver-dispatch.mjs new file mode 100644 index 000000000..3ea9cac6a --- /dev/null +++ b/scripts/ucr-study-driver-dispatch.mjs @@ -0,0 +1,57 @@ +#!/usr/bin/env node +/** Dispatch one standard study request to its configured cross-CLI driver. */ + +import { spawnSync } from 'node:child_process'; +import { existsSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { + STUDY_DRIVER_PROTOCOL, + studyDriverChildEnvironment, + studyDriverProcessTimeoutMs, + studyDirectionEnvironmentKey, +} from '../ucr/index.mjs'; + +const chunks = []; +let size = 0; +for await (const chunk of process.stdin) { + size += chunk.length; + if (size > 1024 * 1024) throw new Error('study driver request exceeds 1 MiB'); + chunks.push(chunk); +} +const raw = Buffer.concat(chunks).toString('utf8'); +const request = JSON.parse(raw); +const trial = request?.trial; +const key = studyDirectionEnvironmentKey( + trial?.producerClient, + trial?.consumerClient +); +const bundledDriver = fileURLToPath( + new URL('./ucr-live-study-driver.mjs', import.meta.url) +); +const command = + process.env[key] || process.env.UCR_STUDY_DIRECTION_DRIVER || bundledDriver; +if (!command || !existsSync(command)) { + process.stderr.write( + `${key} or UCR_STUDY_DIRECTION_DRIVER must identify an executable ${STUDY_DRIVER_PROTOCOL} driver\n` + ); + process.exitCode = 2; +} else { + const moduleDriver = /\.(?:cjs|mjs|js)$/i.test(command); + const child = spawnSync( + moduleDriver ? process.execPath : command, + moduleDriver ? [command] : [], + { + cwd: process.cwd(), + env: studyDriverChildEnvironment(process.env), + input: raw, + encoding: 'utf8', + timeout: studyDriverProcessTimeoutMs(trial, process.env), + maxBuffer: 64 * 1024 * 1024, + windowsHide: true, + shell: false, + } + ); + process.stdout.write(String(child.stdout || '')); + process.stderr.write(String(child.stderr || child.error?.message || '')); + process.exitCode = child.status ?? 1; +} diff --git a/scripts/verify-live-dashboard.mjs b/scripts/verify-live-dashboard.mjs new file mode 100644 index 000000000..eeeda5b74 --- /dev/null +++ b/scripts/verify-live-dashboard.mjs @@ -0,0 +1,89 @@ +#!/usr/bin/env node +/** Read-only Playwright verification against an already running dashboard. */ + +import { mkdirSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { chromium } from 'playwright'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +const base = process.argv[2] || 'http://localhost:3101'; +const output = join(ROOT, 'artifacts', 'live-dashboard'); +mkdirSync(output, { recursive: true }); + +const browser = await chromium.launch(); +const page = await browser.newPage({ viewport: { width: 1440, height: 900 } }); +const errors = []; +page.on('console', (message) => { + if (message.type() === 'error') errors.push(`console:${message.text()}`); +}); +page.on('pageerror', (error) => errors.push(`page:${error.message}`)); +page.on('requestfailed', (request) => + errors.push(`request:${request.url()}:${request.failure()?.errorText}`) +); + +try { + await page.goto(base, { waitUntil: 'networkidle' }); + const overviewBody = await page.locator('body').innerText(); + const overview = { + title: await page.title(), + heading: await page.locator('.brand').first().innerText(), + claudeHardcode: /what claude did/i.test(overviewBody), + genericActivity: + /which actions cost the most|what it has learned|agent activity/i.test( + overviewBody + ), + }; + await page.screenshot({ + path: join(output, 'overview.png'), + fullPage: true, + }); + + await page.goto(`${base}/wiki`, { waitUntil: 'networkidle' }); + await page.waitForSelector('#wiki-list li'); + const host = page.locator('#wiki-graph-3d'); + const graph3d = await host.evaluate((element) => ({ + width: element.clientWidth, + height: element.clientHeight, + projected: + element.__knowledgeGraph3d?.projectedNodes?.().length ?? null, + selected: element.dataset.selected || null, + })); + await page.locator('#mode-constellation').click(); + await page.waitForTimeout(800); + await page.locator('#wiki-list li').first().click(); + await page.waitForTimeout(500); + const afterSelect = await host.evaluate((element) => ({ + projected: + element.__knowledgeGraph3d?.projectedNodes?.().length ?? null, + selected: element.dataset.selected || null, + })); + const wiki = { + listCount: await page.locator('#wiki-list li').count(), + graph3d, + afterSelect, + detailVisible: await page.locator('#wiki-detail').isVisible(), + }; + await page.screenshot({ + path: join(output, 'wiki-3d.png'), + fullPage: true, + }); + + const passed = + !overview.claudeHardcode && + overview.genericActivity && + wiki.listCount > 0 && + wiki.graph3d.width > 0 && + wiki.graph3d.height > 0 && + wiki.graph3d.projected > 0 && + wiki.afterSelect.projected > 0 && + Boolean(wiki.afterSelect.selected) && + wiki.detailVisible && + errors.length === 0; + process.stdout.write( + `${JSON.stringify({ passed, base, overview, wiki, errors }, null, 2)}\n` + ); + if (!passed) process.exitCode = 1; +} finally { + await browser.close(); +} diff --git a/scripts/verify-ucr-qualification.mjs b/scripts/verify-ucr-qualification.mjs new file mode 100644 index 000000000..b49a75ac6 --- /dev/null +++ b/scripts/verify-ucr-qualification.mjs @@ -0,0 +1,156 @@ +#!/usr/bin/env node +/** Recompute bounded qualification gates only from hash-bound signed evidence. */ + +import { existsSync, readFileSync, readdirSync } from 'node:fs'; +import { basename, dirname, isAbsolute, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { + canonicalJson, + sha256, + studyQualificationVerdict, + validateModelAttestation, + verifyEvidenceLedger, +} from '../ucr/index.mjs'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +const RESULTS = join(ROOT, 'evals', 'ucr', 'results'); +const TRUST = join(ROOT, 'evals', 'ucr', 'trusted-evidence-keys'); +const requireQualified = process.argv.includes('--require-qualified'); +const configured = process.argv + .slice(2) + .filter((value) => value !== '--require-qualified'); +const files = configured.length + ? configured.map((value) => (isAbsolute(value) ? value : resolve(value))) + : readdirSync(RESULTS) + .filter((name) => /^full-study-qualification-.*\.json$/i.test(name)) + .map((name) => join(RESULTS, name)); +const results = []; + +for (const path of files.sort()) { + const report = JSON.parse(readFileSync(path, 'utf8')); + const { reportHash, ...body } = report; + const rows = report?.ledger?.rows || []; + const keyId = String(report?.ledgerKeyId || ''); + const trustedKeyPath = /^[A-Za-z0-9._-]+$/.test(keyId) + ? join(TRUST, `${keyId}.pem`) + : null; + const reportHashValid = sha256(body) === reportHash; + let trustedPublicKey = null; + if (trustedKeyPath && existsSync(trustedKeyPath)) { + try { + trustedPublicKey = readFileSync(trustedKeyPath, 'utf8'); + } catch { + trustedPublicKey = null; + } + } + const ledgerVerification = verifyEvidenceLedger(report.ledger, { + publicKey: trustedPublicKey, + }); + const signatureTrusted = + Boolean(trustedPublicKey) && ledgerVerification.validSignature === true; + const selectedTrialIdsHashValid = + report?.selection?.selectedTrialIdsHash === + sha256(rows.map((row) => row.trialId)); + const observedArms = [ + ...new Set(rows.map((row) => String(row?.arm || '')).filter(Boolean)), + ].sort(); + const declaredArms = String(report?.selection?.arms || '') + .split(',') + .map((arm) => arm.trim()) + .filter(Boolean) + .sort(); + const armsMatchRows = + observedArms.length > 0 && + canonicalJson(declaredArms) === canonicalJson(observedArms); + const selectedTrialsMatchRows = + Number.isInteger(report?.selection?.selectedTrials) && + report.selection.selectedTrials === rows.length; + const qualification = report?.selection + ? studyQualificationVerdict({ + rows, + selectedTrialCount: rows.length, + plannedArms: observedArms, + failures: report.failures, + }) + : { + status: 'unverifiable', + passed: false, + failed: ['selectionMetadataMissing'], + }; + const embeddedQualificationMatches = + canonicalJson(report?.qualification ?? null) === + canonicalJson(qualification); + const modelAttestationsValid = + rows.length > 0 && + rows.every((row) => { + const attestations = row?.modelAttestations || []; + if (attestations.length !== row?.expectedProviderInvocations) return false; + return attestations.every((entry) => { + const producer = entry?.role === 'producer'; + const expectedClient = producer + ? row.producerClient + : row.consumerClient; + const expectedModel = producer + ? row.producerModelVersion + : row.consumerModelVersion; + const expectedTransport = producer + ? row.producerTransport + : row.consumerTransport; + return ( + entry?.transport === expectedTransport && + validateModelAttestation(entry?.attestation, { + client: expectedClient, + requestedModel: expectedModel, + providerRequestId: entry?.providerRequestId, + }).valid + ); + }); + }); + const consumerMcpIsolation = + rows.length > 0 && + rows.every((row) => row?.consumerMcpExposed === false); + const semanticEvidenceVerified = + rows.length > 0 && + rows.every( + (row) => + row?.semanticEvidenceVerification?.verified === true && + Array.isArray(row.semanticEvidenceVerification.anchors) && + row.semanticEvidenceVerification.anchors.length > 0 + ); + const artifactValid = + reportHashValid && + ledgerVerification.valid && + signatureTrusted && + selectedTrialIdsHashValid && + armsMatchRows && + selectedTrialsMatchRows && + embeddedQualificationMatches; + const qualified = + artifactValid && + modelAttestationsValid && + consumerMcpIsolation && + semanticEvidenceVerified && + qualification.passed; + const passed = artifactValid && (!requireQualified || qualified); + results.push({ + file: basename(path), + passed, + artifactValid, + qualified, + reportHashValid, + ledgerValid: ledgerVerification.valid, + signatureTrusted, + selectedTrialIdsHashValid, + armsMatchRows, + selectedTrialsMatchRows, + modelAttestationsValid, + consumerMcpIsolation, + semanticEvidenceVerified, + qualification, + embeddedQualificationMatches, + }); +} + +const passed = results.length > 0 && results.every((result) => result.passed); +process.stdout.write(canonicalJson({ passed, requireQualified, results })); +if (!passed) process.exitCode = 1; diff --git a/scripts/verify-ucr-study-readiness.mjs b/scripts/verify-ucr-study-readiness.mjs new file mode 100644 index 000000000..90c31e8bd --- /dev/null +++ b/scripts/verify-ucr-study-readiness.mjs @@ -0,0 +1,158 @@ +#!/usr/bin/env node +/** CI preflight proving that the frozen program can produce every release metric. */ + +import { existsSync, readFileSync, writeFileSync } from 'node:fs'; +import { dirname, join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { + buildFullStudyPlan, + BENCHMARK_ARMS, + BENCHMARK_FAMILIES, + canonicalJson, + freezeBenchmark, + preRegisterBenchmark, + releaseMetricCoveragePreflight, + sha256, + STUDY_DRIVER_PROTOCOL, + STUDY_NEGATIVE_ARMS, + studyDriverRegistry, + UCR_CLIENT_REGISTRY, + zeroFailureWilsonSampleSize, +} from '../ucr/index.mjs'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +const OUTPUT = join(ROOT, 'evals', 'ucr', 'results', 'study-readiness-v1.json'); +const write = process.argv.includes('--write'); +const source = JSON.parse( + readFileSync(join(ROOT, 'evals', 'ucr', 'benchmark-v1.json'), 'utf8') +); +const benchmark = freezeBenchmark(source); +const registration = preRegisterBenchmark(source, source.pilot); +const clients = JSON.parse( + readFileSync(join(ROOT, 'evals', 'ucr', 'study-clients-v1.json'), 'utf8') +).clients; +const competitorKinds = JSON.parse( + readFileSync(join(ROOT, 'evals', 'ucr', 'competitors-v1.json'), 'utf8') +).baselines.map((baseline) => baseline.kind); +const pairedRepetitionsPerCell = Math.ceil( + registration.powerAnalysis.perArm / benchmark.tasks.length +); +const negativeRepetitionsPerCell = Math.ceil( + zeroFailureWilsonSampleSize({ + comparisons: clients.length ** 2 * STUDY_NEGATIVE_ARMS.length, + }).samples / benchmark.tasks.length +); +const plan = buildFullStudyPlan({ + benchmark, + clients, + secret: 'ci-structure-only-secret', + registration, + pairedRepetitionsPerCell, + negativeRepetitionsPerCell, + minimumSubgroupPairs: pairedRepetitionsPerCell, +}); +const coverage = releaseMetricCoveragePreflight({ + plan, + competitorKinds, + productionStages: [ + 'shadow-selection', + 'observe-only', + 'advisory-canary', + 'verification-canary', + 'scoped-enforcement', + ], +}); +const driverRegistry = studyDriverRegistry(); +const driverContractPassed = + Object.keys(driverRegistry).length === + Object.keys(UCR_CLIENT_REGISTRY).length && + Object.values(driverRegistry).every( + (driver) => driver.protocol === STUDY_DRIVER_PROTOCOL + ); +const powered = + plan.primaryPairsPerDirection >= registration.powerAnalysis.perArm && + plan.negativeSamplesPerDirectionPerArm >= + plan.minimumNegativeSamplesPerDirection; +const familyCount = new Set(plan.trials.map((trial) => trial.family)).size; +const armCount = new Set(plan.trials.map((trial) => trial.arm)).size; +const readinessChecks = { + ...coverage.design.checks, + benchmarkFamilies: familyCount >= BENCHMARK_FAMILIES.length, + benchmarkArms: armCount >= BENCHMARK_ARMS.length, +}; +const designPassed = + coverage.passed && + powered && + driverContractPassed && + Object.values(readinessChecks).every(Boolean) && + coverage.missingMetrics.length === 0; +const body = { + schemaVersion: 'ucr.study-readiness/1', + passed: designPassed, + structuralPlanHash: plan.planHash, + benchmarkHash: benchmark.manifestHash, + registrationHash: registration.registrationHash, + pairedRepetitionsPerCell, + primaryPairsPerDirection: plan.primaryPairsPerDirection, + negativeRepetitionsPerCell, + negativeSamplesPerDirectionPerArm: plan.negativeSamplesPerDirectionPerArm, + negativeConfidence: plan.negativeConfidence, + trials: plan.trials.length, + providerInvocations: plan.trials.reduce((sum, trial) => { + if ( + !Number.isInteger(trial.expectedProviderInvocations) || + trial.expectedProviderInvocations < 2 + ) + throw new Error( + `trial ${trial.trialId} omits expectedProviderInvocations` + ); + return sum + trial.expectedProviderInvocations; + }, 0), + mappedMetrics: Object.keys(coverage.metricSources).length, + representativeStudyClients: clients.length, + universalDriverClients: Object.keys(driverRegistry).length, + driverContractPassed, + missingMetrics: coverage.missingMetrics, + coverage: { + ...coverage.design.coverage, + familyCount, + requiredFamilies: BENCHMARK_FAMILIES.length, + armCount, + requiredArms: BENCHMARK_ARMS.length, + }, + checks: readinessChecks, +}; +const report = { ...body, reportHash: sha256(body) }; +if (write) + writeFileSync(OUTPUT, `${JSON.stringify(report, null, 2)}\n`, 'utf8'); +let artifactValid = false; +if (existsSync(OUTPUT)) { + try { + artifactValid = + canonicalJson(JSON.parse(readFileSync(OUTPUT, 'utf8'))) === + canonicalJson(report); + } catch { + artifactValid = false; + } +} +const passed = designPassed && artifactValid; +console.log( + canonicalJson({ + passed, + artifactValid, + output: OUTPUT, + planHash: plan.planHash, + pairedRepetitionsPerCell, + primaryPairsPerDirection: plan.primaryPairsPerDirection, + negativeRepetitionsPerCell, + negativeSamplesPerDirectionPerArm: plan.negativeSamplesPerDirectionPerArm, + trials: plan.trials.length, + providerInvocations: body.providerInvocations, + mappedMetrics: Object.keys(coverage.metricSources).length, + representativeStudyClients: clients.length, + universalDriverClients: Object.keys(driverRegistry).length, + missingMetrics: coverage.missingMetrics, + coverage: body.coverage, + }) +); +if (!passed) process.exitCode = 1; diff --git a/scripts/verify-wiki-interactions.mjs b/scripts/verify-wiki-interactions.mjs index 22eaed348..a53b2533a 100644 --- a/scripts/verify-wiki-interactions.mjs +++ b/scripts/verify-wiki-interactions.mjs @@ -31,14 +31,18 @@ const SHOTS = join(ROOT, 'artifacts', 'wiki-interactions'); const results = []; const check = (name, pass, detail = '') => { results.push({ name, pass }); - console.log(`${pass ? ' PASS' : ' FAIL'} ${name}${detail ? ` -- ${detail}` : ''}`); + console.log( + `${pass ? ' PASS' : ' FAIL'} ${name}${detail ? ` -- ${detail}` : ''}` + ); }; async function waitForServer(attempts = 40) { for (let i = 0; i < attempts; i++) { try { if ((await fetch(`${BASE}/api/wiki/status`)).ok) return true; - } catch { /* not up */ } + } catch { + /* not up */ + } await new Promise((r) => setTimeout(r, 500)); } return false; @@ -46,20 +50,37 @@ async function waitForServer(attempts = 40) { async function main() { mkdirSync(SHOTS, { recursive: true }); - cpSync(join(ROOT, 'src', 'dashboard', 'public'), join(ROOT, 'dist', 'dashboard', 'public'), { recursive: true }); + cpSync( + join(ROOT, 'src', 'dashboard', 'public'), + join(ROOT, 'dist', 'dashboard', 'public'), + { recursive: true } + ); // Seed the hostile graph into this run's isolated directory. await new Promise((resolve, reject) => { - const seed = spawn(process.execPath, [join(ROOT, 'scripts', 'seed-edge-cases.mjs'), GRAPH], - { cwd: ROOT, stdio: 'ignore', windowsHide: true, env: { ...process.env } }); - seed.on('exit', (code) => (code === 0 ? resolve() : reject(new Error(`seed exited ${code}`)))); + const seed = spawn( + process.execPath, + [join(ROOT, 'scripts', 'seed-edge-cases.mjs'), GRAPH], + { cwd: ROOT, stdio: 'ignore', windowsHide: true, env: { ...process.env } } + ); + seed.on('exit', (code) => + code === 0 ? resolve() : reject(new Error(`seed exited ${code}`)) + ); seed.on('error', reject); }); - const server = spawn(process.execPath, [join(ROOT, 'dist', 'server', 'web-server.js')], - { cwd: ROOT, stdio: 'ignore', windowsHide: true, env: { ...process.env, PORT: String(PORT) } }); + const server = spawn( + process.execPath, + [join(ROOT, 'dist', 'server', 'web-server.js')], + { + cwd: ROOT, + stdio: 'ignore', + windowsHide: true, + env: { ...process.env, PORT: String(PORT) }, + } + ); - if (!await waitForServer()) { + if (!(await waitForServer())) { console.error(`server never came up on ${BASE}`); server.kill(); process.exit(1); @@ -76,10 +97,17 @@ async function main() { const errors = []; try { - const page = await browser.newPage({ viewport: { width: 1440, height: 900 } }); - page.on('console', (m) => { if (m.type() === 'error') errors.push(m.text()); }); + const page = await browser.newPage({ + viewport: { width: 1440, height: 900 }, + }); + page.on('console', (m) => { + if (m.type() === 'error') errors.push(m.text()); + }); page.on('pageerror', (e) => errors.push(`pageerror: ${e.message}`)); - page.on('dialog', async (d) => { errors.push(`unexpected dialog: ${d.message()}`); await d.dismiss(); }); + page.on('dialog', async (d) => { + errors.push(`unexpected dialog: ${d.message()}`); + await d.dismiss(); + }); await page.goto(`${BASE}/wiki`, { waitUntil: 'networkidle' }); await page.waitForSelector('#wiki-list li'); @@ -88,13 +116,28 @@ async function main() { const titleBefore = await page.title(); check('no payload executed on load', titleBefore !== 'PWNED', titleBefore); - check('no injected element from a claim', await page.locator('img[onerror]').count() === 0); - check('no injected script tag', await page.locator('.wiki-list script').count() === 0); + check( + 'no injected element from a claim', + (await page.locator('img[onerror]').count()) === 0 + ); + check( + 'no injected script tag', + (await page.locator('.wiki-list script').count()) === 0 + ); // The payload must be VISIBLE as text -- escaped, not stripped. Silently // dropping it would also "pass" while hiding real content from the user. - const payloadShown = await page.locator('#wiki-list .wiki-claim', { hasText: 'a claim containing markup' }).first().innerText(); - check('markup in a claim renders as literal text', payloadShown.includes('']) { + for (const hostile of [ + '*', + '(', + '[', + '\\', + '.*', + '?', + '${x}', + '', + ]) { + const errorsBefore = errors.length; await search.fill(hostile); await page.waitForTimeout(220); - check(`search survives ${JSON.stringify(hostile)}`, errors.length === 0, errors[0] || ''); + check( + `search survives ${JSON.stringify(hostile)}`, + errors.length === errorsBefore, + errors.slice(errorsBefore).join(' | ') + ); } await search.fill(''); await page.waitForTimeout(400); - check('clearing search restores the list', await page.locator('#wiki-list li').count() > 1); + check( + 'clearing search restores the list', + (await page.locator('#wiki-list li').count()) > 1 + ); /* ---- Type filter -------------------------------------------------- */ - for (const value of ['finding', 'decision', 'failure', 'command', 'map', '']) { + await page.selectOption('#wiki-type', ''); + await page.waitForTimeout(300); + const allIds = await page + .locator('#wiki-list li') + .evaluateAll((items) => items.map((item) => item.dataset.id)); + + for (const value of ['finding', 'decision', 'failure', 'command', 'map']) { await page.selectOption('#wiki-type', value); await page.waitForTimeout(300); - const count = await page.locator('#wiki-list li').count(); - check(`type filter "${value || 'all'}" applies`, count >= 0, `${count} results`); + const actualIds = await page + .locator('#wiki-list li') + .evaluateAll((items) => items.map((item) => item.dataset.id)); + const expectedIds = await page.evaluate(async (type) => { + const result = await ( + await fetch( + `/api/wiki/search?type=${encodeURIComponent(type)}&limit=50` + ) + ).json(); + return result.items.map((item) => item.id); + }, value); + check( + `type filter "${value}" shows exactly that API slice`, + actualIds.length <= allIds.length && + actualIds.length === expectedIds.length && + actualIds.every((id, index) => id === expectedIds[index]), + `${actualIds.length} of ${allIds.length}` + ); } + await page.selectOption('#wiki-type', ''); + await page.waitForTimeout(300); + const restoredIds = await page + .locator('#wiki-list li') + .evaluateAll((items) => items.map((item) => item.dataset.id)); + check( + 'clearing the type filter restores the unfiltered list', + restoredIds.length === allIds.length && + restoredIds.every((id, index) => id === allIds[index]), + `${restoredIds.length} of ${allIds.length}` + ); + /* ---- Selecting a finding ------------------------------------------ */ await page.locator('#wiki-list li').first().click(); - await page.waitForSelector('#wiki-graph .wiki-node'); - check('clicking a finding renders the focus graph', - await page.locator('#wiki-graph .wiki-node').count() > 0); - check('the detail drawer opens', await page.locator('#wiki-detail').isVisible()); - check('the page reserves space rather than being covered', - await page.evaluate(() => document.body.classList.contains('wiki-detail-open'))); + await page.waitForSelector('#wiki-detail:visible'); + check( + 'clicking a finding selects it in the 3D graph', + Boolean( + await page.locator('#wiki-graph-3d').getAttribute('data-selected') + ) + ); + check( + 'the 3D graph remains the active view after selection', + (await page + .locator('#mode-constellation') + .getAttribute('aria-pressed')) === 'true' + ); + check( + 'the detail drawer opens', + await page.locator('#wiki-detail').isVisible() + ); + check( + 'the page reserves space rather than being covered', + await page.evaluate(() => + document.body.classList.contains('wiki-detail-open') + ) + ); // The controls the drawer used to sit on top of must still be clickable. - check('mode buttons remain clickable with the drawer open', - await page.locator('#mode-constellation').isEnabled()); + check( + 'mode buttons remain clickable with the drawer open', + await page.locator('#mode-constellation').isEnabled() + ); await page.locator('#mode-constellation').click({ timeout: 5000 }); await page.waitForTimeout(1200); - check('constellation renders with the drawer open', - await page.locator('#wiki-graph .wiki-node').count() > 0); + check( + '3D graph renders with the drawer open', + Number(await page.locator('#wiki-graph-3d').getAttribute('data-nodes')) > + 0 + ); await page.locator('#mode-focus').click(); await page.waitForTimeout(600); + await page.waitForSelector('#wiki-graph .wiki-node'); + check( + 'explicit Focus renders the one-hop graph', + (await page.locator('#wiki-graph .wiki-node').count()) > 0 + ); /* ---- Keyboard paths ----------------------------------------------- */ await page.keyboard.press('Escape'); - check('Escape closes the drawer', !(await page.locator('#wiki-detail').isVisible())); + check( + 'Escape closes the drawer', + !(await page.locator('#wiki-detail').isVisible()) + ); const item = page.locator('#wiki-list li').nth(1); await item.focus(); await page.keyboard.press('Enter'); await page.waitForTimeout(500); - check('Enter on a focused list item selects it', await page.locator('#wiki-detail').isVisible()); + check( + 'Enter on a focused list item selects it', + await page.locator('#wiki-detail').isVisible() + ); await page.keyboard.press('Escape'); const node = page.locator('#wiki-graph .wiki-node').first(); await node.focus(); await page.keyboard.press('Enter'); await page.waitForTimeout(500); - check('Enter on a graph node re-centres', await page.locator('#wiki-detail').isVisible()); + check( + 'Enter on a graph node re-centres', + await page.locator('#wiki-detail').isVisible() + ); await page.keyboard.press('Escape'); /* ---- Graph navigation --------------------------------------------- */ @@ -176,38 +315,67 @@ async function main() { // Assert the centre becomes THE NODE CLICKED, not merely that it changed. // "Changed" cannot distinguish a correct re-centre from a re-render that // happened to land elsewhere, and it fails opaquely when it does. - const centreId = () => page.evaluate(() => - document.querySelector('#wiki-graph .wiki-node.is-center')?.dataset.id || ''); + const centreId = () => + page.evaluate( + () => + document.querySelector('#wiki-graph .wiki-node.is-center')?.dataset + .id || '' + ); + + const neighbourId = await page.evaluate( + () => + document.querySelector('#wiki-graph .wiki-node:not(.is-center)') + ?.dataset.id || '' + ); - const neighbourId = await page.evaluate(() => - document.querySelector('#wiki-graph .wiki-node:not(.is-center)')?.dataset.id || ''); + check('the focus graph renders a neighbour node', Boolean(neighbourId)); if (neighbourId) { - await page.locator(`#wiki-graph .wiki-node[data-id="${neighbourId}"]`).click(); - await page.waitForFunction( - (id) => document.querySelector('#wiki-graph .wiki-node.is-center')?.dataset.id === id, - neighbourId, { timeout: 5000 }, - ).catch(() => {}); - check('clicking a neighbour re-centres on THAT node', - await centreId() === neighbourId, `${await centreId()} vs ${neighbourId}`); + await page + .locator(`#wiki-graph .wiki-node[data-id="${neighbourId}"]`) + .click(); + await page + .waitForFunction( + (id) => + document.querySelector('#wiki-graph .wiki-node.is-center')?.dataset + .id === id, + neighbourId, + { timeout: 5000 } + ) + .catch(() => {}); + check( + 'clicking a neighbour re-centres on THAT node', + (await centreId()) === neighbourId, + `${await centreId()} vs ${neighbourId}` + ); } // A retired finding is still REACHABLE via supersedes edges. It must never // render like a live one -- that is the same failure as serving a stale // finding bare. const retiredMarked = await page.evaluate(async () => { - const found = await (await fetch('/api/wiki/search?q=corrected&limit=5')).json(); + const found = await ( + await fetch('/api/wiki/search?q=corrected&limit=5') + ).json(); const item = found.items[0]; if (!item) return 'no corrected finding'; - const detail = await (await fetch(`/api/wiki/node/${encodeURIComponent(item.id)}`)).json(); + const detail = await ( + await fetch(`/api/wiki/node/${encodeURIComponent(item.id)}`) + ).json(); const superseded = detail.neighbours.find((n) => n.kind === 'finding'); if (!superseded) return 'no superseded neighbour'; return superseded.retired === true ? 'marked' : 'UNMARKED'; }); - check('a retired finding reached through the graph is marked as retired', - retiredMarked === 'marked', retiredMarked); - - await page.screenshot({ path: join(SHOTS, 'focus-hostile.png'), fullPage: true }); + check( + 'a retired finding reached through the graph is marked as retired', + retiredMarked === 'marked', + retiredMarked + ); + + await page.screenshot({ + path: join(SHOTS, 'focus-hostile.png'), + fullPage: true, + }); /* ---- Curation ------------------------------------------------------ */ @@ -218,29 +386,42 @@ async function main() { const countBefore = await page.locator('#wiki-list li').count(); await page.locator('#detail-correct').click(); await page.waitForTimeout(600); - check('an empty correction is refused', - await page.locator('#wiki-list li').count() === countBefore); - - await page.locator('#detail-claim').fill('a correction typed through the UI'); + check( + 'an empty correction is refused', + (await page.locator('#wiki-list li').count()) === countBefore + ); + + await page + .locator('#detail-claim') + .fill('a correction typed through the UI'); await page.locator('#detail-correct').click(); await page.waitForTimeout(900); - const corrected = await page.locator('.wiki-list li', { hasText: 'a correction typed through the UI' }).count(); + const corrected = await page + .locator('.wiki-list li', { + hasText: 'a correction typed through the UI', + }) + .count(); check('a correction round-trips and appears in the list', corrected > 0); await page.locator('#wiki-list li').first().click(); await page.waitForSelector('#wiki-detail:visible'); await page.locator('#detail-pin').click(); await page.waitForTimeout(800); - check('pinning succeeds and the list refreshes', await page.locator('#wiki-list li').count() > 0); + check( + 'pinning succeeds and the list refreshes', + (await page.locator('#wiki-list li').count()) > 0 + ); await page.locator('#wiki-list li').first().click(); await page.waitForSelector('#wiki-detail:visible'); const beforeRetire = await page.locator('#wiki-list li').count(); await page.locator('#detail-retire').click(); await page.waitForTimeout(900); - check('retiring removes the finding from the list', - await page.locator('#wiki-list li').count() < beforeRetire, - `${beforeRetire} -> ${await page.locator('#wiki-list li').count()}`); + check( + 'retiring removes the finding from the list', + (await page.locator('#wiki-list li').count()) < beforeRetire, + `${beforeRetire} -> ${await page.locator('#wiki-list li').count()}` + ); /* ---- Audit tab ------------------------------------------------------ */ @@ -250,41 +431,67 @@ async function main() { check('audit surfaces contradictions', /Contradicted/i.test(audit)); check('audit surfaces unanchored findings', /Unanchored/i.test(audit)); check('audit surfaces low confidence', /Low confidence/i.test(audit)); - check('the audit badge shows a count', - Number(await page.locator('#audit-count').innerText()) > 0); + check( + 'the audit badge shows a count', + Number(await page.locator('#audit-count').innerText()) > 0 + ); await page.locator('#audit-groups .wiki-list li').first().click(); await page.waitForTimeout(700); - check('an audit entry opens its detail', await page.locator('#wiki-detail').isVisible()); + check( + 'an audit entry opens its detail', + await page.locator('#wiki-detail').isVisible() + ); await page.keyboard.press('Escape'); - await page.screenshot({ path: join(SHOTS, 'audit-hostile.png'), fullPage: true }); + await page.screenshot({ + path: join(SHOTS, 'audit-hostile.png'), + fullPage: true, + }); await page.locator('.wiki-tab[data-tab="explore"]').click(); await page.waitForTimeout(400); - check('switching back to Explore restores the list', - await page.locator('#wiki-list li').count() > 0); + check( + 'switching back to Explore restores the list', + (await page.locator('#wiki-list li').count()) > 0 + ); /* ---- Export --------------------------------------------------------- */ const exported = await (await fetch(`${BASE}/api/wiki/export`)).text(); check('export contains findings', exported.includes('##')); - check('export excludes retired findings', !exported.includes('withdrawn by a human')); + check( + 'export excludes retired findings', + !exported.includes('withdrawn by a human') + ); check('export labels human-asserted entries', exported.includes('human')); /* ---- The mutating route rejects cross-site callers -------------------- */ const noHeader = await fetch(`${BASE}/api/wiki/curate`, { - method: 'POST', headers: { 'content-type': 'application/json' }, + method: 'POST', + headers: { 'content-type': 'application/json' }, body: JSON.stringify({ action: 'retire', key: 'f-normal' }), }); - check('curate without the header is refused', noHeader.status === 403, String(noHeader.status)); + check( + 'curate without the header is refused', + noHeader.status === 403, + String(noHeader.status) + ); const crossOrigin = await fetch(`${BASE}/api/wiki/curate`, { method: 'POST', - headers: { 'content-type': 'application/json', 'x-token-optimizer': 'dashboard', origin: 'http://evil.example' }, + headers: { + 'content-type': 'application/json', + 'x-token-optimizer': 'dashboard', + origin: 'http://evil.example', + }, body: JSON.stringify({ action: 'retire', key: 'f-normal' }), }); - check('curate from another origin is refused', crossOrigin.status === 403, String(crossOrigin.status)); + check( + 'curate from another origin is refused', + crossOrigin.status === 403, + String(crossOrigin.status) + ); /* ---- API robustness --------------------------------------------------- */ @@ -297,33 +504,61 @@ async function main() { ['traversal attempt', '/api/wiki/search?project=../../../../etc'], ]) { const res = await fetch(BASE + path); - check(`API handles ${label}`, res.status === 200 || res.status === 404, String(res.status)); + check( + `API handles ${label}`, + res.status === 200 || res.status === 404, + String(res.status) + ); } // The removed parameter must not resurrect a caller-chosen path. - const traversal = await (await fetch(`${BASE}/api/wiki/status?project=${encodeURIComponent('C:/Windows')}`)).json(); - check('project param cannot redirect the graph directory', traversal.dir === GRAPH, traversal.dir); + const traversal = await ( + await fetch( + `${BASE}/api/wiki/status?project=${encodeURIComponent('C:/Windows')}` + ) + ).json(); + check( + 'project param cannot redirect the graph directory', + traversal.dir === GRAPH, + traversal.dir + ); /* ---- Narrow viewport --------------------------------------------------- */ - const narrow = await browser.newPage({ viewport: { width: 480, height: 900 } }); - narrow.on('pageerror', (e) => errors.push(`narrow pageerror: ${e.message}`)); + const narrow = await browser.newPage({ + viewport: { width: 480, height: 900 }, + }); + narrow.on('pageerror', (e) => + errors.push(`narrow pageerror: ${e.message}`) + ); await narrow.goto(`${BASE}/wiki`, { waitUntil: 'networkidle' }); await narrow.waitForSelector('#wiki-list li'); const overflow = await narrow.evaluate(() => { const el = document.scrollingElement; return { scroll: el.scrollWidth, client: el.clientWidth }; }); - check('no horizontal scroll at 480px', overflow.scroll <= overflow.client + 1, - `${overflow.scroll} vs ${overflow.client}`); + check( + 'no horizontal scroll at 480px', + overflow.scroll <= overflow.client + 1, + `${overflow.scroll} vs ${overflow.client}` + ); await narrow.locator('#wiki-list li').first().click(); await narrow.waitForTimeout(700); - check('the drawer is usable at 480px', await narrow.locator('#wiki-detail').isVisible()); - await narrow.screenshot({ path: join(SHOTS, 'narrow-hostile.png'), fullPage: true }); + check( + 'the drawer is usable at 480px', + await narrow.locator('#wiki-detail').isVisible() + ); + await narrow.screenshot({ + path: join(SHOTS, 'narrow-hostile.png'), + fullPage: true, + }); await narrow.close(); - check('no console or page errors across the whole run', errors.length === 0, - errors.slice(0, 3).join(' | ')); + check( + 'no console or page errors across the whole run', + errors.length === 0, + errors.slice(0, 3).join(' | ') + ); await page.close(); } finally { @@ -334,7 +569,9 @@ async function main() { } const failed = results.filter((r) => !r.pass); - console.log(`\n${results.length - failed.length}/${results.length} checks passed`); + console.log( + `\n${results.length - failed.length}/${results.length} checks passed` + ); console.log(`screenshots: ${SHOTS}`); process.exit(failed.length ? 1 : 0); } diff --git a/scripts/verify-wiki-ui.mjs b/scripts/verify-wiki-ui.mjs index 53d83a570..b5f3b1feb 100644 --- a/scripts/verify-wiki-ui.mjs +++ b/scripts/verify-wiki-ui.mjs @@ -16,13 +16,27 @@ import { chromium } from 'playwright'; import { spawn } from 'node:child_process'; -import { mkdirSync, writeFileSync, rmSync, cpSync, mkdtempSync } from 'node:fs'; +import { + mkdirSync, + writeFileSync, + readFileSync, + rmSync, + cpSync, + mkdtempSync, +} from 'node:fs'; import { join, dirname } from 'node:path'; import { tmpdir } from 'node:os'; import { fileURLToPath } from 'node:url'; const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); const SHOTS = join(ROOT, 'artifacts', 'wiki-ui'); +const evidenceIndex = JSON.parse( + readFileSync( + join(ROOT, 'evals', 'ucr', 'results', 'evidence-index-v2.json'), + 'utf8' + ) +); +const expectedArtifactCount = evidenceIndex.artifacts.length; /** * An isolated port, chosen at random and asserted free. * @@ -215,14 +229,25 @@ async function seed() { for (const arm of ['empty', 'natural', 'oracle', 'irrelevant', 'stale']) { const prevented = ['natural', 'oracle'].includes(arm); record(GRAPH, { - kind: 'handoff-run', pairId: `ui-handoff-${pair}`, - scenarioId: 'verification-entry-point', arm, - producer: { client: 'codex', model: 'gpt-5.6-sol', captureSuccess: true }, + kind: 'handoff-run', + pairId: `ui-handoff-${pair}`, + scenarioId: 'verification-entry-point', + arm, + producer: { + client: 'codex', + model: 'gpt-5.6-sol', + captureSuccess: true, + }, consumer: { - client: 'claude-code', model: 'claude-sonnet-5', correct: true, - firstPass: prevented, mistakeAttempted: !prevented, - mistakeExecuted: !prevented, totalTokens: prevented ? 800 : 1000, - toolCalls: prevented ? 5 : 7, failedToolCalls: prevented ? 0 : 1, + client: 'claude-code', + model: 'claude-sonnet-5', + correct: true, + firstPass: prevented, + mistakeAttempted: !prevented, + mistakeExecuted: !prevented, + totalTokens: prevented ? 800 : 1000, + toolCalls: prevented ? 5 : 7, + failedToolCalls: prevented ? 0 : 1, }, delivery: { beforeFirstExecutedMistake: arm === 'natural' }, }); @@ -230,13 +255,23 @@ async function seed() { } for (const arm of ['empty', 'natural']) { record(GRAPH, { - kind: 'concurrency-run', pairId: 'ui-concurrent-1', arm, - writerCount: 3, captureSuccesses: 3, + kind: 'concurrency-run', + pairId: 'ui-concurrent-1', + arm, + writerCount: 3, + captureSuccesses: 3, integrity: { zeroLoss: true, parseable: true, orphanedFindings: 0 }, - delivery: { expected: arm === 'natural' ? 3 : 0, delivered: arm === 'natural' ? 3 : 0 }, + delivery: { + expected: arm === 'natural' ? 3 : 0, + delivered: arm === 'natural' ? 3 : 0, + }, consumer: { - client: 'claude-code', model: 'claude-sonnet-5', correct: arm === 'empty', - firstPass: false, mistakeAttempted: true, mistakeExecuted: true, + client: 'claude-code', + model: 'claude-sonnet-5', + correct: arm === 'empty', + firstPass: false, + mistakeAttempted: true, + mistakeExecuted: true, }, }); } @@ -331,6 +366,18 @@ async function main() { overflow.scroll <= overflow.client + 1, `${overflow.scroll} vs ${overflow.client}` ); + if (label === 'narrow') { + const splitTracks = await page + .locator('.wiki-split') + .evaluate((split) => + getComputedStyle(split).gridTemplateColumns.trim().split(/\s+/) + ); + check( + 'narrow: knowledge layout stacks into one column', + splitTracks.length === 1, + splitTracks.join(' | ') + ); + } if (label === 'desktop') { const stats = await page.textContent('#graph-stats'); @@ -373,8 +420,158 @@ async function main() { auditCount ); - // Focus mode: clicking a finding must actually draw a graph. - await page.click('.wiki-list li'); + // The default view is a real, navigable 3D projection. A previous test + // only counted SVG nodes, so a flat and nearly blank graph passed. + await page.waitForFunction( + () => + Number(document.querySelector('#wiki-graph-3d')?.dataset.frame) > 0, + null, + { timeout: 5000 } + ); + const graph3d = await page.evaluate(() => { + const host = document.getElementById('wiki-graph-3d'); + const canvas = host.querySelector('canvas'); + const context = canvas.getContext('2d'); + const pixels = context.getImageData( + 0, + 0, + canvas.width, + canvas.height + ).data; + const colours = new Set(); + for (let index = 0; index < pixels.length; index += 256) { + colours.add( + `${pixels[index]},${pixels[index + 1]},${pixels[index + 2]}` + ); + } + const projected = host.__knowledgeGraph3d.projectedNodes(); + const xs = projected.map((node) => node.x); + const ys = projected.map((node) => node.y); + return { + renderer: host.dataset.renderer, + nodes: Number(host.dataset.nodes), + edges: Number(host.dataset.edges), + frame: Number(host.dataset.frame), + colours: colours.size, + spread: + projected.length < 2 + ? 0 + : Math.max( + Math.max(...xs) - Math.min(...xs), + Math.max(...ys) - Math.min(...ys) + ), + }; + }); + check( + '3D graph uses the perspective renderer', + graph3d.renderer === 'perspective-3d', + graph3d.renderer + ); + check( + '3D graph renders real nodes', + graph3d.nodes > 0, + `${graph3d.nodes} nodes` + ); + check( + '3D graph renders real edges', + graph3d.edges > 0, + `${graph3d.edges} edges` + ); + check( + '3D graph paints a non-blank scene', + graph3d.colours > 8, + `${graph3d.colours} sampled colours` + ); + check( + '3D projection spreads nodes through the scene', + graph3d.spread > 60, + `${Math.round(graph3d.spread)}px extent` + ); + + const graphCanvas = page.locator( + '#wiki-graph-3d .knowledge-graph-canvas' + ); + check( + '3D graph exposes an interactive accessibility role', + (await graphCanvas.getAttribute('role')) === 'application' + ); + await graphCanvas.focus(); + await page.keyboard.press('Alt+ArrowRight'); + const keyboardSelection = await page + .locator('#wiki-graph-3d') + .getAttribute('data-selected'); + check( + 'keyboard selection moves to a 3D node', + Boolean(keyboardSelection), + keyboardSelection || '' + ); + await page.keyboard.press('Enter'); + await page.waitForSelector('#wiki-detail:not([hidden])'); + check( + 'keyboard selection opens the node record', + await page.isVisible('#wiki-detail') + ); + await page.click('#detail-close'); + + const graphHost = page.locator('#wiki-graph-3d'); + await graphHost.scrollIntoViewIfNeeded(); + const graphBox = await graphHost.boundingBox(); + const yawBefore = Number(await graphHost.getAttribute('data-yaw')); + const zoomBefore = Number(await graphHost.getAttribute('data-zoom')); + await page.mouse.move( + graphBox.x + graphBox.width * 0.35, + graphBox.y + graphBox.height * 0.52 + ); + await page.mouse.down(); + await page.mouse.move( + graphBox.x + graphBox.width * 0.64, + graphBox.y + graphBox.height * 0.34, + { steps: 10 } + ); + await page.mouse.up(); + await page.mouse.move( + graphBox.x + graphBox.width / 2, + graphBox.y + graphBox.height / 2 + ); + await page.mouse.wheel(0, -360); + await page.waitForTimeout(200); + const yawAfter = Number(await graphHost.getAttribute('data-yaw')); + const zoomAfter = Number(await graphHost.getAttribute('data-zoom')); + check( + 'dragging orbits the 3D camera', + Math.abs(yawAfter - yawBefore) > 0.2, + `${yawBefore.toFixed(2)} -> ${yawAfter.toFixed(2)}` + ); + check( + 'scrolling zooms the 3D camera', + zoomAfter > zoomBefore, + `${zoomBefore.toFixed(2)} -> ${zoomAfter.toFixed(2)}` + ); + + await page.click('#wiki-graph-3d .graph-3d-reset'); + await page.waitForTimeout(100); + const target = await page.evaluate( + () => + document + .getElementById('wiki-graph-3d') + .__knowledgeGraph3d.projectedNodes()[0] + ); + await page.mouse.click(graphBox.x + target.x, graphBox.y + target.y); + await page.waitForSelector('#wiki-detail:not([hidden])'); + const selected3d = await graphHost.getAttribute('data-selected'); + check( + 'clicking a 3D node selects its record', + selected3d === target.id, + selected3d + ); + await page.screenshot({ + path: join(SHOTS, 'constellation.png'), + fullPage: true, + }); + + // Focus mode remains available for a readable one-hop explanation. + await page.click('#detail-close'); + await page.click('#mode-focus'); await page.waitForSelector('#wiki-graph .wiki-node', { timeout: 5000 }); const focusNodes = await page.$$eval( '#wiki-graph .wiki-node', @@ -493,82 +690,25 @@ async function main() { `${smallest.toFixed(1)}px tall` ); - // Constellation. - await page.click('#mode-constellation'); - await page.waitForTimeout(1200); - const constellationNodes = await page.$$eval( - '#wiki-graph .wiki-node', - (n) => n.length - ); - check( - 'constellation renders nodes', - constellationNodes > 0, - `${constellationNodes} nodes` - ); - - const spread = await page.evaluate(() => { - const boxes = [ - ...document.querySelectorAll('#wiki-graph .wiki-node circle'), - ].map((c) => c.getBoundingClientRect()); - if (boxes.length < 2) return 1; - // If the layout failed, every node sits on the same point. - const xs = boxes.map((b) => b.x); - const ys = boxes.map((b) => b.y); - return Math.max( - Math.max(...xs) - Math.min(...xs), - Math.max(...ys) - Math.min(...ys) - ); - }); - check( - 'constellation actually spreads nodes out', - spread > 60, - `${Math.round(spread)}px extent` + // Returning to 3D must preserve a functional renderer after focus and + // drawer reflow, not recreate a blank surface. + await page.click('#detail-close'); + const frameBeforeRoundTrip = Number( + await page.getAttribute('#wiki-graph-3d', 'data-frame') ); - - const constellationBounds = await page.evaluate(() => { - const svg = document - .getElementById('wiki-graph') - .getBoundingClientRect(); - const texts = [ - ...document.querySelectorAll('#wiki-graph .wiki-node text'), - ].map((t) => t.getBoundingClientRect()); - let collisions = 0; - for (let i = 0; i < texts.length; i++) { - for (let j = i + 1; j < texts.length; j++) { - const a = texts[i]; - const b = texts[j]; - if ( - a.left < b.right && - b.left < a.right && - a.top < b.bottom && - b.top < a.bottom - ) - collisions++; - } - } - return { - collisions, - clipped: texts.filter( - (t) => t.right > svg.right + 2 || t.left < svg.left - 2 - ).length, - }; - }); - check( - 'constellation labels stay on the canvas', - constellationBounds.clipped === 0, - `${constellationBounds.clipped} clipped` + await page.click('#mode-constellation'); + await page.waitForFunction( + (baseline) => + Number(document.querySelector('#wiki-graph-3d')?.dataset.frame) > + baseline, + frameBeforeRoundTrip, + { timeout: 5000 } ); check( - 'constellation labels do not collide', - constellationBounds.collisions === 0, - `${constellationBounds.collisions} overlaps` + '3D graph survives focus-mode round trip', + await page.isVisible('#wiki-graph-3d canvas') ); - await page.screenshot({ - path: join(SHOTS, 'constellation.png'), - fullPage: true, - }); - // Audit tab. await page.click('.wiki-tab[data-tab="audit"]'); await page.waitForTimeout(400); @@ -612,9 +752,7 @@ async function main() { /claude-code/i.test(transferText || '') && /gates passed/i.test(transferText || '') ); - const concurrencyText = await page.textContent( - '#evidence-concurrency' - ); + const concurrencyText = await page.textContent('#evidence-concurrency'); check( 'evidence renders concurrent writer integrity and later delivery', /100%/.test(concurrencyText || '') && @@ -647,17 +785,30 @@ async function main() { ); check( 'UCR dashboard renders signed artifacts and live directions', - /Evidence artifacts\s*12\/12/i.test(ucrText || '') && + new RegExp( + `Evidence artifacts\\s*${expectedArtifactCount}/${expectedArtifactCount}`, + 'i' + ).test(ucrText || '') && /Live directions\s*2\/3/i.test(ucrText || '') && /Consumer MCP schema\s*0 tokens max/i.test(ucrText || '') && - /Combined token reduction\s*21\.06%/i.test(ucrText || '') && - /Combined latency reduction\s*18\.36%/i.test(ucrText || '') && - /Known mistake recurrence\s*1 control → 0 runtime/i.test( + /Combined token reduction\s*5\.66%/i.test(ucrText || '') && + /Combined latency reduction\s*-4\.73%/i.test(ucrText || '') && + /Known mistake recurrence\s*0 control → 0 runtime/i.test( ucrText || '' ) && - /Native guard denials\s*1/i.test(ucrText || '') && + /Native guard denials\s*0/i.test(ucrText || '') && /Capture model calls\s*0 additional max/i.test(ucrText || '') ); + check( + 'UCR dashboard separates frozen design from observed evidence', + /Frozen study design\s*54,054 trials \/ 113,022 calls/i.test( + ucrText || '' + ) && + /Release metrics mapped\s*37/i.test(ucrText || '') && + /Universal CLI drivers\s*16 protocol-mapped \/ 3 in powered live matrix/i.test( + ucrText || '' + ) + ); const tierCount = await page.locator('#ucr-tiers tbody tr').count(); const artifactCount = await page .locator('#ucr-artifacts tbody tr') @@ -669,7 +820,7 @@ async function main() { ); check( 'UCR dashboard exposes every integrity-checked study', - artifactCount === 12, + artifactCount === expectedArtifactCount, String(artifactCount) ); check( @@ -691,6 +842,108 @@ async function main() { await page.close(); } + // Overview contract: exercise the real browser renderer against the exact + // object/event shapes returned by the session API. This catches regressions + // where `toolName` became the useless fallback "action" and an object-shaped + // toolBreakdown silently rendered as an empty table. + const overview = await browser.newPage({ + viewport: { width: 1440, height: 900 }, + }); + overview.on('console', (message) => { + if (message.type() === 'error') + consoleErrors.push(`overview: ${message.text()}`); + }); + overview.on('pageerror', (error) => + consoleErrors.push(`overview: ${error.message}`) + ); + await overview.route('**/api/session-summary*', (route) => + route.fulfill({ + contentType: 'application/json', + body: JSON.stringify({ + success: true, + totalTokens: 910, + totalTurns: 3, + totalTools: 2, + duration: '4m', + tokensByCategory: { + tools: { tokens: 910, percent: '100.00' }, + }, + tokensByServer: {}, + toolBreakdown: { + Read: { count: 1, tokens: 610 }, + Grep: { count: 1, tokens: 300 }, + }, + }), + }) + ); + await overview.route('**/api/session-events*', (route) => + route.fulfill({ + contentType: 'application/json', + body: JSON.stringify({ + success: true, + events: [ + { + type: 'tool_call', + toolName: 'Read', + estimatedTokens: 610, + timestamp: '2026-08-10T15:00:00Z', + }, + { + type: 'tool_call', + toolName: 'Grep', + estimatedTokens: 300, + timestamp: '2026-08-10T15:01:00Z', + }, + ], + }), + }) + ); + await overview.goto(`${BASE}/`, { waitUntil: 'networkidle' }); + await overview.waitForSelector('#constellation canvas'); + const overviewText = await overview.textContent('body'); + check( + 'overview is provider and CLI neutral', + !/claude(?: code)?/i.test(overviewText || '') + ); + const eventText = await overview.locator('.event').first().textContent(); + check( + 'overview names the observed tool and token cost', + /Read/.test(eventText || '') && + /610 context tokens/.test(eventText || ''), + eventText?.trim() + ); + check( + 'overview never falls back to generic action rows', + !(await overview.locator('.ev-name').allTextContents()).some( + (name) => name.trim().toLowerCase() === 'action' + ) + ); + const breakdownRows = await overview + .locator('#tool-breakdown-body tr') + .count(); + check( + 'overview renders object-shaped tool totals', + breakdownRows === 2, + `${breakdownRows} rows` + ); + const actionCostRows = await overview.locator('#server-chart .legend li'); + check( + 'overview charts object-shaped action costs instead of showing an empty state', + (await actionCostRows.count()) === 2 && + /Read/.test((await actionCostRows.first().textContent()) || ''), + `${await actionCostRows.count()} chart rows` + ); + check( + 'overview embeds the same 3D graph renderer', + (await overview.getAttribute('#constellation', 'data-renderer')) === + 'perspective-3d' + ); + await overview.screenshot({ + path: join(SHOTS, 'overview.png'), + fullPage: true, + }); + await overview.close(); + check( 'no console or page errors', consoleErrors.length === 0, diff --git a/src/dashboard/public/css/styles.css b/src/dashboard/public/css/styles.css index 9c0ad60ea..4f7d8f653 100644 --- a/src/dashboard/public/css/styles.css +++ b/src/dashboard/public/css/styles.css @@ -24,7 +24,11 @@ .hero-saved { background: - radial-gradient(120% 100% at 0% 0%, rgba(46, 230, 168, 0.09), transparent 62%), + radial-gradient( + 120% 100% at 0% 0%, + rgba(46, 230, 168, 0.09), + transparent 62% + ), var(--surface); display: flex; flex-direction: column; @@ -143,11 +147,18 @@ .constellation { flex: 1; min-height: 190px; - display: flex; - align-items: center; + position: relative; + overflow: hidden; + border: 1px solid var(--line); + border-radius: 11px; margin: 4px 0 10px; } +.constellation .graph-3d-help, +.constellation .graph-3d-reset { + display: none; +} + .node-dot { animation: breathe 5.5s ease-in-out infinite; animation-delay: calc(var(--i) * -0.42s); @@ -327,7 +338,7 @@ section.card { .event { display: grid; - grid-template-columns: 8px 1fr auto auto; + grid-template-columns: 8px minmax(180px, 1fr) auto auto; align-items: center; gap: 12px; padding: 8px 10px; @@ -347,6 +358,7 @@ section.card { } .ev-name { + display: block; font-family: var(--mono); font-size: 12.5px; overflow: hidden; @@ -354,6 +366,27 @@ section.card { white-space: nowrap; } +.ev-copy { + min-width: 0; +} + +.ev-detail { + display: block; + color: var(--ink-3); + font-size: 11.5px; + margin-top: 1px; +} + +@media (max-width: 700px) { + .event { + grid-template-columns: 8px 1fr auto; + } + + .ev-meta { + display: none; + } +} + .ev-meta { color: var(--saved); font-size: 12px; diff --git a/src/dashboard/public/css/tokens.css b/src/dashboard/public/css/tokens.css index 635a5fad1..67e40027b 100644 --- a/src/dashboard/public/css/tokens.css +++ b/src/dashboard/public/css/tokens.css @@ -398,3 +398,88 @@ a:focus-visible, transition: none !important; } } + +/* ------------------------------------------------ interactive 3D graph -- */ + +.knowledge-graph-3d { + position: relative; + width: 100%; + height: 100%; + min-height: inherit; + overflow: hidden; + isolation: isolate; +} + +.knowledge-graph-canvas { + display: block; + width: 100%; + height: 100%; + cursor: grab; + touch-action: none; +} + +.knowledge-graph-canvas.is-dragging { + cursor: grabbing; +} + +.graph-3d-chrome { + position: absolute; + inset: 12px 12px auto; + display: flex; + align-items: center; + gap: 10px; + pointer-events: none; + z-index: 2; +} + +.graph-3d-badge { + border: 1px solid rgba(46, 230, 168, 0.28); + border-radius: 999px; + background: rgba(10, 15, 20, 0.72); + color: var(--saved); + font-size: 10.5px; + font-weight: 700; + letter-spacing: 0.08em; + padding: 5px 9px; + text-transform: uppercase; + backdrop-filter: blur(8px); +} + +.graph-3d-help { + color: var(--ink-2); + font-size: 11.5px; + text-shadow: 0 1px 3px var(--bg); +} + +.graph-3d-reset { + margin-left: auto; + border: 1px solid rgba(163, 182, 196, 0.22); + border-radius: 999px; + background: rgba(10, 15, 20, 0.72); + color: var(--ink-2); + font: 550 11.5px var(--font); + padding: 5px 10px; + pointer-events: auto; + cursor: pointer; + backdrop-filter: blur(8px); +} + +.graph-3d-reset:hover { + border-color: var(--saved); + color: var(--ink); +} + +.graph-3d-tooltip { + position: absolute; + z-index: 3; + width: min(230px, 70%); + border: 1px solid var(--line-bright); + border-radius: 7px; + background: rgba(10, 15, 20, 0.92); + box-shadow: var(--shadow); + color: var(--ink); + font-size: 11.5px; + line-height: 1.45; + padding: 7px 9px; + pointer-events: none; +} diff --git a/src/dashboard/public/css/wiki.css b/src/dashboard/public/css/wiki.css index 83e7a740d..1db560ca4 100644 --- a/src/dashboard/public/css/wiki.css +++ b/src/dashboard/public/css/wiki.css @@ -12,8 +12,25 @@ /* ---------------------------------------------------------------- intro -- */ .intro { + position: relative; margin-bottom: 26px; - max-width: 68ch; + max-width: none; + overflow: hidden; + border: 1px solid var(--line); + border-radius: var(--radius); + background: + radial-gradient( + 80% 160% at 100% 0%, + rgba(122, 162, 247, 0.16), + transparent 62% + ), + radial-gradient( + 70% 120% at 0% 100%, + rgba(46, 230, 168, 0.08), + transparent 68% + ), + var(--surface); + padding: 28px 30px; } .intro h1 { @@ -26,6 +43,7 @@ } .intro p { + max-width: 74ch; margin: 0; color: var(--ink-2); font-size: 14.5px; @@ -231,12 +249,13 @@ .wiki-split { display: grid; - grid-template-columns: minmax(0, 400px) 1fr; + grid-template-columns: minmax(280px, 0.72fr) minmax(520px, 1.45fr); gap: var(--gap); align-items: start; } -@media (max-width: 900px) { +/* Stack before the 280px + 520px tracks, gap, and page padding can overflow. */ +@media (max-width: 920px) { .wiki-split { grid-template-columns: 1fr; } @@ -308,16 +327,33 @@ background: var(--surface); border: 1px solid var(--line); border-radius: var(--radius); - padding: 18px; - min-height: 420px; + padding: 14px; + min-height: 590px; display: flex; flex-direction: column; } -#wiki-graph { +.wiki-graph-stage { + position: relative; width: 100%; flex: 1; - min-height: 320px; + min-height: 460px; + overflow: hidden; + border: 1px solid var(--line); + border-radius: 11px; + background: #0a1017; +} + +#wiki-graph, +#wiki-graph-3d { + width: 100%; + height: 100%; + min-height: 460px; +} + +#wiki-graph-3d[hidden], +#wiki-graph[hidden] { + display: none; } .wiki-node text { @@ -337,7 +373,7 @@ } .wiki-graph-hint { - margin: 10px 0 0; + margin: 10px 2px 0; } .wiki-legend { @@ -495,6 +531,53 @@ z-index: 60; } +.wiki-detail h3 { + margin: 20px 0 18px; + font-size: 18px; + line-height: 1.45; + letter-spacing: -0.01em; +} + +.wiki-detail dl { + display: grid; + grid-template-columns: 112px minmax(0, 1fr); + gap: 8px 14px; + margin: 0; + border-top: 1px solid var(--line); + border-bottom: 1px solid var(--line); + padding: 15px 0; +} + +.wiki-detail dt { + color: var(--ink-3); + font-size: 12px; +} + +.wiki-detail dd { + min-width: 0; + margin: 0; + color: var(--ink-2); + font-size: 12.5px; + overflow-wrap: anywhere; +} + +#detail-claim { + width: 100%; + min-height: 88px; + resize: vertical; + background: var(--bg); + border: 1px solid var(--line-bright); + border-radius: var(--radius-sm); + color: var(--ink); + font: 13px/1.5 var(--font); + padding: 10px 12px; +} + +#detail-claim:focus { + outline: 2px solid var(--saved); + outline-offset: 1px; +} + body.wiki-detail-open { overflow: hidden; } diff --git a/src/dashboard/public/index.html b/src/dashboard/public/index.html index 01baef6ae..415cb02dc 100644 --- a/src/dashboard/public/index.html +++ b/src/dashboard/public/index.html @@ -37,14 +37,18 @@

Saved so far -

@@ -57,12 +61,16 @@ — estimated - @@ -81,13 +89,16 @@

What it has learned -

@@ -105,7 +116,11 @@

Where the context went -

-

Bigger slices are the best places to save more.

+

+ Bigger slices are the best places to save more. +

- Which tools cost the most -

-

Ranked by how much of the workspace each one used.

+

+ Ranked by how much of the workspace each one used. +

-

What Claude did

-

Most recent actions first.

+

Recent agent activity

+

+ What the active CLI actually did, most recent first. +

@@ -156,8 +182,8 @@

- Updated never. Everything here is measured - on this machine and never leaves it. + Updated never. Everything here is + measured on this machine and never leaves it.

diff --git a/src/dashboard/public/js/charts.js b/src/dashboard/public/js/charts.js index cce76a798..c6ee49ba8 100644 --- a/src/dashboard/public/js/charts.js +++ b/src/dashboard/public/js/charts.js @@ -134,7 +134,7 @@ function emptyRing() { const wrap = document.createElement('div'); wrap.className = 'chart-blank'; wrap.innerHTML = - '

Nothing measured yet. This fills in as Claude works.

'; + '

Nothing measured yet. This fills in as your coding agents work.

'; return wrap; } diff --git a/src/dashboard/public/js/dashboard.js b/src/dashboard/public/js/dashboard.js index 443496ce2..56072803e 100644 --- a/src/dashboard/public/js/dashboard.js +++ b/src/dashboard/public/js/dashboard.js @@ -14,10 +14,11 @@ */ import { donut, comparison, compact, fmt, escapeHtml } from './charts.js'; +import { createKnowledgeGraph3D } from './graph3d.js'; const API = '/api'; -/** Published Claude input rate. Shown as a scale, never as an invoice. */ +/** Reference rate for scale only. Provider and model prices vary. */ const USD_PER_MILLION_TOKENS = 3; const CATEGORY_COLORS = { @@ -28,10 +29,10 @@ const CATEGORY_COLORS = { }; const CATEGORY_LABELS = { - tools: 'Tools Claude ran', + tools: 'Agent tool calls', hooks: 'Automatic checks', system_reminders: 'Background notes', - responses: 'Claude’s replies', + responses: 'Agent responses', }; const SERVER_COLORS = [ @@ -46,6 +47,7 @@ const SERVER_COLORS = [ ]; const $ = (id) => document.getElementById(id); +let overviewGraph = null; document.addEventListener('DOMContentLoaded', () => { $('refresh-btn').addEventListener('click', load); @@ -107,7 +109,7 @@ function renderSavings(balance, status) { foot.innerHTML = learned ? `Still measuring. ${fmt(learned)} things learned so far — the savings ` + `figure appears once there is enough of a comparison to be honest about.` - : `Nothing measured yet. Work normally in Claude Code for a few minutes ` + + : `Nothing measured yet. Work normally in a supported coding agent for a few minutes ` + `and this fills itself in.`; return; } @@ -195,6 +197,8 @@ function renderGraph(constellation, status) { const remembered = Number(status?.nodes) || 0; if (!nodes.length) { + overviewGraph?.destroy(); + overviewGraph = null; // TWO DIFFERENT EMPTIES, and conflating them produced a page that // contradicted itself: the verdict said "across 475 things it has learned" // while this panel said "Nothing learned yet". The constellation surfaces @@ -204,20 +208,26 @@ function renderGraph(constellation, status) { ? '
◍
' + `

${fmt(remembered)} files remembered

` + '

Nothing has been concluded about them yet. Findings appear here as ' + - 'Claude works out how the pieces fit together.

' + 'your coding agents work out how the pieces fit together.

' : '
◌
' + '

Nothing learned yet

' + - '

As Claude reads your code, each file is remembered here so it never ' + - 'has to be read twice.

' + - '

Open a file in Claude Code, then refresh.

'; + '

As a supported agent reads your code, each file is remembered here so it does not ' + + 'have to be read twice.

' + + '

Open a file with your coding agent, then refresh.

'; // The counts are still true and still worth showing. stats.innerHTML = remembered ? statRows(status) : ''; return; } - host.innerHTML = ''; - host.appendChild(nodeField(nodes)); + if (!overviewGraph) { + overviewGraph = createKnowledgeGraph3D(host, constellation, { + compact: true, + cap: 60, + }); + } else { + overviewGraph.update(constellation); + } stats.innerHTML = statRows(status); } @@ -317,7 +327,7 @@ function renderKpis(s) { host.innerHTML = '
◔
' + '

No active session

' + - '

These fill in while you are working in Claude Code. The savings above ' + + '

These fill in while you are working in a supported coding agent. The savings above ' + 'are kept per project, so they stay accurate between sessions.

' + '

Start a session, then refresh.

'; return; @@ -328,7 +338,7 @@ function renderKpis(s) { label: 'Context used', value: compact(s.totalTokens), tip: - 'How much of Claude’s limited workspace this session has taken up. ' + + 'How much of the active agent’s limited workspace this session has taken up. ' + 'The smaller this is for the same work, the better.', }, { @@ -340,7 +350,7 @@ function renderKpis(s) { label: 'Actions taken', value: fmt(s.totalTools), tip: - 'Reads, edits, searches and commands Claude ran on your behalf. ' + + 'Reads, edits, searches and commands the active agent ran on your behalf. ' + 'Each one is a chance to save context.', }, { @@ -370,7 +380,7 @@ function renderCategories(s) { if (!s?.tokensByCategory) { host.innerHTML = teach( 'Nothing to break down yet', - 'Once Claude runs a few actions, this shows which kinds of thing filled the workspace.' + 'Once a coding agent runs a few actions, this shows which kinds of thing filled the workspace.' ); return; } @@ -387,13 +397,17 @@ function renderCategories(s) { function renderServers(s) { const host = $('server-chart'); - const entries = Object.entries(s?.tokensByServer || {}).filter( + let entries = Object.entries(s?.tokensByServer || {}).filter( ([, v]) => (v?.tokens || v) > 0 ); + if (!entries.length) + entries = toolRows(s) + .map((row) => [row.name || row.tool, row.tokens]) + .filter(([name, tokens]) => name && Number(tokens) > 0); if (!entries.length) { host.innerHTML = teach( - 'No tool servers yet', - 'When Claude uses a connected tool, its share of the workspace shows up here.' + 'No measured actions yet', + 'When a coding agent reads, edits, searches, or runs a command, its share of the workspace shows up here.' ); return; } @@ -415,33 +429,64 @@ function renderTimeline(events) { if (!events?.length) { host.innerHTML = teach( 'Nothing has happened yet', - 'Every read, edit and command Claude runs will appear here in order, newest first.' + 'Every observed read, edit, search, and command will appear here in order, newest first.' ); return; } host.innerHTML = events .slice(0, 40) - .map( - (e) => ` + .map((e) => { + const action = describeEvent(e); + return `
- ${escapeHtml(e.name || e.tool || 'action')} - ${e.tokens ? `${compact(e.tokens)} tokens` : ''} + + ${escapeHtml(action.tool)} + ${escapeHtml(action.detail)} + + ${action.tokens ? `${compact(action.tokens)} context tokens` : 'no context recorded'} ${e.timestamp ? new Date(e.timestamp).toLocaleTimeString() : ''} -
` - ) + `; + }) .join(''); } +function describeEvent(event) { + const tool = String( + event.toolName || event.name || event.tool || event.type || 'agent event' + ); + const details = { + Edit: 'Edited project content', + Write: 'Wrote project content', + Read: 'Read project context', + Bash: 'Ran a terminal command', + Shell: 'Ran a terminal command', + Grep: 'Searched project text', + Glob: 'Matched project files', + TodoWrite: 'Updated the task plan', + WebSearch: 'Searched the web', + BashOutput: 'Inspected terminal output', + KillShell: 'Stopped a background command', + }; + const eventType = String(event.type || 'agent event').replaceAll('_', ' '); + return { + tool, + detail: + details[tool] || + `${eventType.charAt(0).toUpperCase()}${eventType.slice(1)}`, + tokens: Number(event.estimatedTokens ?? event.tokens ?? 0), + }; +} + function renderBreakdown(s) { const body = $('tool-breakdown-body'); - const rows = s?.toolBreakdown || s?.tools || []; + const rows = toolRows(s); if (!rows.length) { body.innerHTML = '' + teach( 'No actions counted yet', - 'Each kind of action Claude takes gets a row here, so repeat costs are easy to spot.' + 'Each kind of observed agent action gets a row here, so repeat costs are easy to spot.' ) + ''; return; @@ -458,6 +503,13 @@ function renderBreakdown(s) { .join(''); } +function toolRows(summary) { + const source = summary?.toolBreakdown || summary?.tools || []; + return Array.isArray(source) + ? source + : Object.entries(source).map(([name, values]) => ({ name, ...values })); +} + /** An empty state that teaches. Never "No data available". */ function teach(title, body) { return ( diff --git a/src/dashboard/public/js/graph3d.js b/src/dashboard/public/js/graph3d.js new file mode 100644 index 000000000..c7150603c --- /dev/null +++ b/src/dashboard/public/js/graph3d.js @@ -0,0 +1,508 @@ +/* + * A small, dependency-free 3D knowledge graph renderer. + * + * The dashboard must work offline, so this uses Canvas 2D as a projection + * surface rather than downloading a WebGL library. The model itself is truly + * three-dimensional: nodes have stable x/y/z coordinates, perspective and + * depth ordering; pointer drag orbits the camera, the wheel zooms, and nodes + * are hit-tested in projected screen space. + */ + +const KIND_COLOURS = { + file: '#7aa2f7', + symbol: '#b48ef7', + finding: '#2ee6a8', + task: '#f0a742', + decision: '#2ee6a8', + failure: '#f2545b', +}; + +function hash(text) { + let value = 2166136261; + for (const char of String(text)) { + value ^= char.charCodeAt(0); + value = Math.imul(value, 16777619); + } + return value >>> 0; +} + +function caption(node) { + const value = + node.claim || + node.name || + String(node.key || '') + .split(/[\\/]/) + .pop(); + return String(value || node.id || 'unknown node'); +} + +function coordinates(node, index, count) { + // Fibonacci sphere plus stable per-id radius jitter. Unlike random layout, + // returning to this project returns every node to the same place. + const seed = hash(node.id || node.key || index); + const y = 1 - ((index + 0.5) / Math.max(1, count)) * 2; + const radiusAtY = Math.sqrt(Math.max(0, 1 - y * y)); + const theta = + index * Math.PI * (3 - Math.sqrt(5)) + (seed % 360) * (Math.PI / 180); + const shell = 1.05 + ((seed >>> 8) % 100) / 260; + return { + x: Math.cos(theta) * radiusAtY * shell, + y: y * shell, + z: Math.sin(theta) * radiusAtY * shell, + }; +} + +function rotate(point, yaw, pitch) { + const cy = Math.cos(yaw); + const sy = Math.sin(yaw); + const cp = Math.cos(pitch); + const sp = Math.sin(pitch); + const x = point.x * cy - point.z * sy; + const z1 = point.x * sy + point.z * cy; + return { + x, + y: point.y * cp - z1 * sp, + z: point.y * sp + z1 * cp, + }; +} + +function roundedRect(ctx, x, y, width, height, radius) { + ctx.beginPath(); + if (typeof ctx.roundRect === 'function') + ctx.roundRect(x, y, width, height, radius); + else ctx.rect(x, y, width, height); + ctx.fill(); +} + +export function createKnowledgeGraph3D(host, initial, options = {}) { + host.innerHTML = ''; + host.classList.add('knowledge-graph-3d'); + host.dataset.renderer = 'perspective-3d'; + + const canvas = document.createElement('canvas'); + canvas.className = 'knowledge-graph-canvas'; + canvas.tabIndex = 0; + canvas.setAttribute('role', 'application'); + canvas.setAttribute( + 'aria-keyshortcuts', + 'Alt+ArrowLeft Alt+ArrowRight Enter Space ArrowLeft ArrowRight ArrowUp ArrowDown Plus Minus R' + ); + host.appendChild(canvas); + + const chrome = document.createElement('div'); + chrome.className = 'graph-3d-chrome'; + chrome.innerHTML = + 'Interactive 3D' + + 'Drag to orbit · scroll to zoom · click a node · Alt+←/→ then Enter by keyboard' + + ''; + host.appendChild(chrome); + + const tooltip = document.createElement('div'); + tooltip.className = 'graph-3d-tooltip'; + tooltip.hidden = true; + host.appendChild(tooltip); + + const context = canvas.getContext('2d', { alpha: false }); + const view = { + yaw: -0.42, + pitch: -0.24, + zoom: options.compact ? 0.9 : 1.08, + velocityYaw: 0, + velocityPitch: 0, + selected: options.selected || null, + hovered: null, + }; + let graph = initial || { nodes: [], edges: [] }; + let points = []; + let projected = []; + let width = 0; + let height = 0; + let frame = 0; + let animation = 0; + let dragging = false; + let moved = false; + let previous = { x: 0, y: 0 }; + let destroyed = false; + + function prepare() { + const nodes = (graph.nodes || []).slice(0, options.cap || 150); + points = nodes.map((node, index) => ({ + node, + ...coordinates(node, index, nodes.length), + })); + if (!points.some((point) => point.node.id === view.selected)) + view.selected = null; + const index = new Map(points.map((point) => [point.node.id, point])); + points.forEach((point) => { + point.degree = 0; + }); + for (const edge of graph.edges || []) { + if (index.has(edge.from)) index.get(edge.from).degree += 1; + if (index.has(edge.to)) index.get(edge.to).degree += 1; + } + canvas.setAttribute( + 'aria-label', + `Interactive 3D knowledge graph with ${points.length} nodes and ${(graph.edges || []).length} connections` + ); + host.dataset.nodes = String(points.length); + host.dataset.edges = String((graph.edges || []).length); + } + + function resize() { + const box = host.getBoundingClientRect(); + const ratio = Math.min(2, window.devicePixelRatio || 1); + width = Math.max(280, Math.round(box.width)); + height = Math.max(options.compact ? 190 : 420, Math.round(box.height)); + canvas.width = Math.round(width * ratio); + canvas.height = Math.round(height * ratio); + canvas.style.width = `${width}px`; + canvas.style.height = `${height}px`; + context.setTransform(ratio, 0, 0, ratio, 0, 0); + draw(); + } + + function project(point) { + const rotated = rotate(point, view.yaw, view.pitch); + const cameraDepth = rotated.z + 4.2; + const focal = Math.min(width, height) * 1.52 * view.zoom; + return { + ...point, + rotated, + screenX: width / 2 + (rotated.x * focal) / cameraDepth, + screenY: height / 2 + (rotated.y * focal) / cameraDepth, + scale: Math.max(0.58, Math.min(1.45, 4.2 / cameraDepth)), + }; + } + + function drawBackdrop() { + const gradient = context.createRadialGradient( + width * 0.5, + height * 0.45, + 0, + width * 0.5, + height * 0.5, + Math.max(width, height) * 0.72 + ); + gradient.addColorStop(0, '#172638'); + gradient.addColorStop(0.52, '#0f1823'); + gradient.addColorStop(1, '#0a1017'); + context.fillStyle = gradient; + context.fillRect(0, 0, width, height); + + // Sparse stable stars make camera movement legible without competing with + // the graph. They are decoration, never data marks. + for (let index = 0; index < 46; index += 1) { + const seed = hash(`background-${index}`); + const x = (seed % 1000) / 1000; + const y = ((seed >>> 10) % 1000) / 1000; + context.fillStyle = `rgba(122, 162, 247, ${0.07 + ((seed >>> 20) % 9) / 100})`; + context.fillRect(x * width, y * height, 1, 1); + } + } + + function draw() { + if (destroyed || !width || !height) return; + drawBackdrop(); + projected = points.map(project).sort((a, b) => b.rotated.z - a.rotated.z); + const byId = new Map(projected.map((point) => [point.node.id, point])); + + context.lineCap = 'round'; + for (const edge of graph.edges || []) { + const from = byId.get(edge.from); + const to = byId.get(edge.to); + if (!from || !to) continue; + const active = + view.selected && + (edge.from === view.selected || edge.to === view.selected); + const depth = (from.scale + to.scale) / 2; + context.strokeStyle = active + ? 'rgba(46, 230, 168, 0.58)' + : `rgba(122, 162, 247, ${Math.min(0.32, 0.1 * depth + 0.06)})`; + context.lineWidth = active ? 1.7 : 0.8; + context.beginPath(); + context.moveTo(from.screenX, from.screenY); + context.lineTo(to.screenX, to.screenY); + context.stroke(); + } + + // Paint farthest-first so nearer nodes, glows, and labels remain visible. + // Label priority runs in the opposite direction without changing paint order. + const nearestRank = new Map( + [...projected].reverse().map((point, index) => [point.node.id, index]) + ); + projected.forEach((point) => { + const rank = nearestRank.get(point.node.id) ?? Infinity; + const selected = point.node.id === view.selected; + const hovered = point.node.id === view.hovered; + const colour = + KIND_COLOURS[point.node.kind] || + KIND_COLOURS[point.node.type] || + '#9fb2c2'; + const baseRadius = point.node.kind === 'finding' ? 6.2 : 4.8; + const radius = + baseRadius * point.scale + Math.min(2.5, point.degree * 0.28); + + if (selected || hovered) { + const glow = context.createRadialGradient( + point.screenX, + point.screenY, + radius, + point.screenX, + point.screenY, + radius * 3.8 + ); + glow.addColorStop( + 0, + selected ? 'rgba(46,230,168,.38)' : 'rgba(122,162,247,.3)' + ); + glow.addColorStop(1, 'rgba(46,230,168,0)'); + context.fillStyle = glow; + context.beginPath(); + context.arc(point.screenX, point.screenY, radius * 3.8, 0, Math.PI * 2); + context.fill(); + } + + context.shadowColor = colour; + context.shadowBlur = selected ? 18 : hovered ? 12 : 5; + context.fillStyle = colour; + context.beginPath(); + context.arc(point.screenX, point.screenY, radius, 0, Math.PI * 2); + context.fill(); + context.shadowBlur = 0; + context.strokeStyle = selected ? '#effff9' : 'rgba(255,255,255,.38)'; + context.lineWidth = selected ? 1.8 : 0.7; + context.stroke(); + + // Keep the field readable: label the selected/hovered nodes and the most + // connected front-facing nodes, not all 150 at once. + const label = selected || hovered || (point.degree > 0 && rank < 8); + if (label && !options.compact) { + const text = caption(point.node).slice( + 0, + selected || hovered ? 54 : 25 + ); + context.font = `${selected ? 600 : 500} 11px -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif`; + const textWidth = Math.min(300, context.measureText(text).width); + const labelX = Math.min( + width - textWidth - 22, + point.screenX + radius + 7 + ); + const labelY = Math.max(18, Math.min(height - 12, point.screenY + 4)); + context.fillStyle = 'rgba(10,15,20,.78)'; + roundedRect(context, labelX - 5, labelY - 13, textWidth + 10, 19, 5); + context.fillStyle = selected ? '#eafff7' : '#b9c9d5'; + context.fillText(text, labelX, labelY, 300); + } + }); + + frame += 1; + host.dataset.frame = String(frame); + host.dataset.yaw = view.yaw.toFixed(4); + host.dataset.pitch = view.pitch.toFixed(4); + host.dataset.zoom = view.zoom.toFixed(4); + host.dataset.selected = view.selected || ''; + host.dispatchEvent( + new CustomEvent('graph3d:render', { detail: { frame } }) + ); + } + + function animate() { + cancelAnimationFrame(animation); + const tick = () => { + view.yaw += view.velocityYaw; + view.pitch = Math.max( + -1.22, + Math.min(1.22, view.pitch + view.velocityPitch) + ); + view.velocityYaw *= 0.91; + view.velocityPitch *= 0.91; + draw(); + if (Math.abs(view.velocityYaw) + Math.abs(view.velocityPitch) > 0.0002) + animation = requestAnimationFrame(tick); + }; + animation = requestAnimationFrame(tick); + } + + function localPoint(event) { + const box = canvas.getBoundingClientRect(); + return { x: event.clientX - box.left, y: event.clientY - box.top }; + } + + function nearestNode(point, radius = 18) { + let hit = null; + let distance = radius; + for (const node of projected) { + const next = Math.hypot(node.screenX - point.x, node.screenY - point.y); + if (next < distance) { + hit = node; + distance = next; + } + } + return hit; + } + + canvas.addEventListener('pointerdown', (event) => { + dragging = true; + moved = false; + previous = localPoint(event); + view.velocityYaw = 0; + view.velocityPitch = 0; + canvas.setPointerCapture(event.pointerId); + canvas.classList.add('is-dragging'); + }); + + canvas.addEventListener('pointermove', (event) => { + const point = localPoint(event); + if (dragging) { + const dx = point.x - previous.x; + const dy = point.y - previous.y; + moved ||= Math.abs(dx) + Math.abs(dy) > 2; + view.velocityYaw = dx * 0.0046; + view.velocityPitch = dy * 0.0046; + view.yaw += view.velocityYaw; + view.pitch = Math.max( + -1.22, + Math.min(1.22, view.pitch + view.velocityPitch) + ); + previous = point; + draw(); + return; + } + const hit = nearestNode(point); + const next = hit?.node.id || null; + if (next !== view.hovered) { + view.hovered = next; + canvas.style.cursor = hit ? 'pointer' : 'grab'; + tooltip.hidden = !hit; + if (hit) { + tooltip.textContent = `${hit.node.kind || 'node'} · ${caption(hit.node)}`; + tooltip.style.left = `${Math.min(width - 230, point.x + 14)}px`; + tooltip.style.top = `${Math.max(12, point.y - 18)}px`; + } + draw(); + } + }); + + const stopDragging = (event) => { + if (!dragging) return; + dragging = false; + canvas.classList.remove('is-dragging'); + const point = localPoint(event); + if (!moved) { + const hit = nearestNode(point); + if (hit) { + view.selected = hit.node.id; + options.onSelect?.(hit.node); + } + } else { + animate(); + } + draw(); + }; + canvas.addEventListener('pointerup', stopDragging); + canvas.addEventListener('pointercancel', stopDragging); + canvas.addEventListener('pointerleave', () => { + if (!dragging) { + view.hovered = null; + tooltip.hidden = true; + draw(); + } + }); + + canvas.addEventListener( + 'wheel', + (event) => { + event.preventDefault(); + view.zoom = Math.max( + 0.55, + Math.min(2.5, view.zoom * Math.exp(-event.deltaY * 0.0012)) + ); + draw(); + }, + { passive: false } + ); + + canvas.addEventListener('keydown', (event) => { + const step = 0.1; + if ( + event.altKey && + (event.key === 'ArrowLeft' || event.key === 'ArrowRight') && + points.length + ) { + const ids = points.map((point) => point.node.id); + const current = ids.indexOf(view.selected); + const direction = event.key === 'ArrowLeft' ? -1 : 1; + const start = current < 0 ? (direction < 0 ? 0 : -1) : current; + view.selected = ids[(start + direction + ids.length) % ids.length]; + canvas.setAttribute( + 'aria-label', + `Selected ${caption(points.find((point) => point.node.id === view.selected).node)} in the interactive 3D knowledge graph. Press Enter to open.` + ); + } else if (event.key === 'Enter' || event.key === ' ') { + const selected = + points.find((point) => point.node.id === view.selected) || points[0]; + if (!selected) return; + view.selected = selected.node.id; + options.onSelect?.(selected.node); + } else if (event.key === 'ArrowLeft') view.yaw -= step; + else if (event.key === 'ArrowRight') view.yaw += step; + else if (event.key === 'ArrowUp') + view.pitch = Math.max(-1.22, view.pitch - step); + else if (event.key === 'ArrowDown') + view.pitch = Math.min(1.22, view.pitch + step); + else if (event.key === '+' || event.key === '=') + view.zoom = Math.min(2.5, view.zoom * 1.12); + else if (event.key === '-' || event.key === '_') + view.zoom = Math.max(0.55, view.zoom / 1.12); + else if (event.key.toLowerCase() === 'r') reset(); + else return; + event.preventDefault(); + draw(); + }); + + function reset() { + view.yaw = -0.42; + view.pitch = -0.24; + view.zoom = options.compact ? 0.9 : 1.08; + view.velocityYaw = 0; + view.velocityPitch = 0; + draw(); + } + + chrome.querySelector('.graph-3d-reset').addEventListener('click', reset); + + const observer = new ResizeObserver(resize); + observer.observe(host); + prepare(); + resize(); + + const api = { + update(next) { + graph = next || { nodes: [], edges: [] }; + prepare(); + draw(); + }, + select(id) { + view.selected = + id && points.some((point) => point.node.id === id) ? id : null; + draw(); + }, + reset, + projectedNodes() { + return projected.map((point) => ({ + id: point.node.id, + x: point.screenX, + y: point.screenY, + })); + }, + destroy() { + destroyed = true; + cancelAnimationFrame(animation); + observer.disconnect(); + host.innerHTML = ''; + delete host.__knowledgeGraph3d; + }, + }; + host.__knowledgeGraph3d = api; + return api; +} diff --git a/src/dashboard/public/js/wiki.js b/src/dashboard/public/js/wiki.js index e13ad707d..23e452ef7 100644 --- a/src/dashboard/public/js/wiki.js +++ b/src/dashboard/public/js/wiki.js @@ -1,3 +1,5 @@ +import { createKnowledgeGraph3D } from './graph3d.js'; + /** * Wiki graph browser. * @@ -6,16 +8,16 @@ * the page fail in exactly the air-gapped and locked-down environments where a * local-first knowledge graph is most wanted. * - * TWO MODES ON ONE CANVAS, because they answer different questions: + * TWO LOCAL-FIRST MODES, because they answer different questions: * * FOCUS -- one node centred with its direct edges radiating out. Readable at * any graph size, because what it renders is bounded by the node's degree * rather than by the size of the graph. * - * CONSTELLATION -- force-directed over a BOUNDED subgraph. The hairball that - * makes force layouts useless comes from running physics over everything; the - * server caps the subgraph, so this stays a navigation map you click into. - * Physics stops as soon as it settles rather than spinning forever. + * EXPLORE IN 3D -- a perspective canvas over a BOUNDED subgraph. Dragging, + * wheel zoom, keyboard orbit, and direct node picking make the learned graph + * navigable without a CDN or a WebGL dependency. The server cap prevents the + * scene from turning into an unbounded hairball. */ const KINDS = ['file', 'symbol', 'finding', 'task']; @@ -41,11 +43,12 @@ const el = (id) => document.getElementById(id); const svgNS = 'http://www.w3.org/2000/svg'; const state = { - mode: 'focus', + mode: 'constellation', offset: 0, selected: null, items: [], }; +let knowledgeGraph3d = null; /** Kind colour, read from CSS so the validated palette has one home. */ function colourFor(kind) { @@ -281,6 +284,64 @@ async function loadUcr() { ['Canonical events', nf.format(status.events)], ['Typed objects', nf.format(status.graph?.objects || 0)], ['Certified clients', nf.format(status.certifiedClients)], + [ + 'Effectiveness verdict', + status.tieredVerdict?.effectiveness?.status || 'insufficient', + ], + [ + 'Superiority verdict', + status.tieredVerdict?.superiority?.status || 'insufficient', + ], + [ + 'Production verdict', + status.tieredVerdict?.production?.status || 'insufficient', + ], + [ + 'Missing effectiveness metrics', + nf.format(status.tieredVerdict?.effectiveness?.missing?.length || 0), + ], + [ + 'Frozen study design', + status.evidenceIndex?.summary.studyDesign?.passed + ? `${nf.format(status.evidenceIndex.summary.studyDesign.trials)} trials / ${nf.format(status.evidenceIndex.summary.studyDesign.providerInvocations)} calls` + : 'not ready', + ], + [ + 'Release metrics mapped', + status.evidenceIndex?.summary.studyDesign?.mappedMetrics != null + ? nf.format(status.evidenceIndex.summary.studyDesign.mappedMetrics) + : 'not measured', + ], + [ + 'Universal CLI drivers', + status.evidenceIndex?.summary.studyDesign?.universalDriverClients != null + ? `${nf.format(status.evidenceIndex.summary.studyDesign.universalDriverClients)} protocol-mapped / ${nf.format(status.evidenceIndex.summary.studyDesign.representativeStudyClients || 0)} in powered live matrix` + : 'not measured', + ], + [ + 'Benchmark family coverage', + status.metrics?.benchmarkFamilyCoverage != null + ? `${(status.metrics.benchmarkFamilyCoverage * 100).toFixed(1)}%` + : 'not measured', + ], + [ + 'Benchmark arm coverage', + status.metrics?.benchmarkArmCoverage != null + ? `${(status.metrics.benchmarkArmCoverage * 100).toFixed(1)}%` + : 'not measured', + ], + [ + 'Worst negative-delivery 95% upper', + status.metrics?.negativeDeliveryIntervalHigh != null + ? `${(status.metrics.negativeDeliveryIntervalHigh * 100).toFixed(2)}%` + : 'not measured', + ], + [ + 'Worst directional token upper', + status.metrics?.directionalTokenOverheadHigh != null + ? `${(status.metrics.directionalTokenOverheadHigh * 100).toFixed(2)}%` + : 'not measured', + ], [ 'Evidence artifacts', status.evidenceIndex @@ -371,7 +432,7 @@ async function loadUcr() { ` ) .join(''); - const verdict = status.verdict?.status || 'insufficient'; + const verdict = status.tieredVerdict?.status || 'insufficient'; const deterministic = status.deterministicEvidence ? `${status.deterministicEvidence.checksPassed}/${status.deterministicEvidence.checksTotal} deterministic gates` : 'no deterministic ledger'; @@ -568,6 +629,8 @@ function drawEdge(svg, x1, y1, x2, y2, kind) { async function renderFocus(nodeId) { const data = await api(`/api/wiki/node/${encodeURIComponent(nodeId)}`); const svg = el('wiki-graph'); + el('wiki-graph-3d').hidden = true; + svg.removeAttribute('hidden'); clearGraph(); el('graph-hint').hidden = true; @@ -607,94 +670,25 @@ async function renderFocus(nodeId) { async function renderConstellation() { const data = await api('/api/wiki/constellation?cap=150'); const svg = el('wiki-graph'); + const host = el('wiki-graph-3d'); + svg.setAttribute('hidden', ''); + host.hidden = false; clearGraph(); - el('graph-hint').hidden = data.nodes.length > 0; - - const { width, height } = paneSize(); - - const positions = new Map(); - data.nodes.forEach((node, index) => { - // Seeded from the index rather than randomly, so a refresh does not - // reshuffle a layout the user has just learned to read. - const angle = index * 2.39996; - const radius = - 20 + - (index / data.nodes.length) * - Math.min(width - LABEL_GUTTER * 2, height) * - 0.42; - positions.set(node.id, { - x: width / 2 + Math.cos(angle) * radius, - y: height / 2 + Math.sin(angle) * radius, + const hint = el('graph-hint'); + hint.hidden = false; + hint.textContent = data.nodes.length + ? 'Drag to orbit, scroll to zoom, and select a node to inspect its evidence and history.' + : 'The 3D knowledge map will appear as supported coding agents capture project findings.'; + + if (knowledgeGraph3d) { + knowledgeGraph3d.update(data); + knowledgeGraph3d.select(state.selected); + } else { + knowledgeGraph3d = createKnowledgeGraph3D(host, data, { + selected: state.selected, + onSelect: (node) => selectNode(node.id), }); - }); - - const REPULSION = 2600; - const SPRING = 0.012; - const IDEAL = 70; - - for (let tick = 0; tick < 160; tick++) { - const cooling = 1 - tick / 160; - - for (const [idA, a] of positions) { - for (const [idB, b] of positions) { - if (idA === idB) continue; - const dx = a.x - b.x; - const dy = a.y - b.y; - const distance = Math.hypot(dx, dy) || 0.01; - const force = (REPULSION / (distance * distance)) * cooling; - a.x += (dx / distance) * force; - a.y += (dy / distance) * force; - } - } - - for (const edge of data.edges) { - const a = positions.get(edge.from); - const b = positions.get(edge.to); - if (!a || !b) continue; - const dx = b.x - a.x; - const dy = b.y - a.y; - const distance = Math.hypot(dx, dy) || 0.01; - const pull = (distance - IDEAL) * SPRING * cooling; - a.x += (dx / distance) * pull; - a.y += (dy / distance) * pull; - b.x -= (dx / distance) * pull; - b.y -= (dy / distance) * pull; - } - - for (const point of positions.values()) { - // Clamped inside the label gutter, not the raw canvas, so a node at the - // boundary still has room for its caption. - point.x = Math.max(LABEL_GUTTER, Math.min(width - LABEL_GUTTER, point.x)); - point.y = Math.max(24, Math.min(height - 24, point.y)); - } } - - // Labels sit on a single baseline per node, so two nodes at a similar height - // overlap even when the MARKS are comfortably apart. The physics has no - // notion of text, so a separate pass nudges colliding captions apart. Without - // it, dense regions produce overlapping unreadable text -- visible only by - // looking at the rendered page, never from the node coordinates alone. - const LABEL_HEIGHT = 14; - const ordered = [...positions.entries()].sort((a, b) => a[1].y - b[1].y); - for (let i = 1; i < ordered.length; i++) { - const [, previous] = ordered[i - 1]; - const [, current] = ordered[i]; - const sameSide = previous.x > width * 0.55 === current.x > width * 0.55; - if (sameSide && current.y - previous.y < LABEL_HEIGHT) { - current.y = Math.min(height - 24, previous.y + LABEL_HEIGHT); - } - } - - for (const edge of data.edges) { - const a = positions.get(edge.from); - const b = positions.get(edge.to); - if (a && b) drawEdge(svg, a.x, a.y, b.x, b.y, null); - } - for (const node of data.nodes) { - const point = positions.get(node.id); - if (point) drawNode(svg, node, point.x, point.y, false); - } - fitLabels(svg); } /** @@ -886,18 +880,24 @@ function setMode(mode) { el('mode-constellation').classList.toggle('is-active', !focus); el('mode-focus').setAttribute('aria-pressed', String(focus)); el('mode-constellation').setAttribute('aria-pressed', String(!focus)); + el('wiki-graph').toggleAttribute('hidden', !focus); + el('wiki-graph-3d').hidden = focus; } async function selectNode(nodeId) { - // Selecting a node IS a focus action, whatever mode was active before. - setMode('focus'); state.selected = nodeId; document .querySelectorAll('.wiki-list li') .forEach((li) => li.setAttribute('aria-current', String(li.dataset.id === nodeId)) ); - await renderFocus(nodeId); + if (state.mode === 'focus') { + await renderFocus(nodeId); + return; + } + knowledgeGraph3d?.select(nodeId); + const data = await api(`/api/wiki/node/${encodeURIComponent(nodeId)}`); + showDetail(data.node); } /* ---- Audit ----------------------------------------------------------- */ @@ -1048,7 +1048,7 @@ if (typeof ResizeObserver !== 'undefined') { lastWidth = width; reRender(); } - }).observe(el('wiki-graph')); + }).observe(el('wiki-graph-stage')); } (async function init() { @@ -1065,6 +1065,7 @@ if (typeof ResizeObserver !== 'undefined') { await Promise.all([ loadBalance(), search(), + renderConstellation(), loadAudit(), loadEvidence(), loadUcr(), diff --git a/src/dashboard/public/wiki.html b/src/dashboard/public/wiki.html index b9751e5b3..64c26ab1a 100644 --- a/src/dashboard/public/wiki.html +++ b/src/dashboard/public/wiki.html @@ -101,20 +101,20 @@

-
+
-
@@ -129,14 +129,20 @@

- - Knowledge graph +
+
+ +

- Pick something on the left to see what it was worked out from. + Drag to orbit, scroll to zoom, and select a node to inspect its + evidence and history.