Skip to content

Benchmark Codex OFF vs MARGINAL with fail-closed evidence - #11

Merged
SignalLayerLabs merged 13 commits into
mainfrom
benchmark/codex-marginal-e2e
Aug 11, 2026
Merged

SignalLayerLabs merged 13 commits into
mainfrom
benchmark/codex-marginal-e2e

Conversation

@SignalLayerLabs

Copy link
Copy Markdown
Owner

Outcome

Adds the audited Codex 0.147 OFF/ON adapter, frozen SWE-bench Lite smoke protocol, six measured inference trajectories, official verifier evidence, and an evidence-first README/site surface.

Measured smoke

  • 3 matched tasks, one repetition per task
  • resolved: 0/3 OFF and 0/3 ON
  • effective tokens: 1,098,747 to 824,839 (24.93% fewer observed)
  • effective latency: 593,106 ms to 565,768 ms (4.61% lower)
  • governance: 0 external tokens, $0, 7,058 ms
  • evaluator status: pass_through; not eligible for a support claim
  • no applied denies in the three real trajectories

Verification

  • authoritative SWE-bench 4.1 Docker verifier: 3/3 completed per lane, zero infrastructure errors
  • Modal audit run 31474500980: symmetric image-build error retained and excluded from scoring
  • infrastructure error_ids and incomplete_ids now fail closed
  • zero verified successes can no longer produce a supported result
  • local gate: 337 tests, Ruff, mypy, protocol validation and JavaScript syntax all green

Scope

This is an exploratory integration smoke, not a general token-saving performance claim. The 20-task repeated canary remains behind the preregistered promotion gate.

@SignalLayerLabs
SignalLayerLabs marked this pull request as ready for review August 11, 2026 09:31
@SignalLayerLabs
SignalLayerLabs merged commit 4c0f8e6 into main Aug 11, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant