Benchmark Codex OFF vs MARGINAL with fail-closed evidence

.dockerignore

@@ -0,0 +1,13 @@
+*
+!pyproject.toml
+!src/
+src/*
+!src/marginal/
+!src/marginal/**
+!benchmark/
+benchmark/*
+!benchmark/__init__.py
+!benchmark/codex_adapter/
+!benchmark/codex_adapter/**
+!benchmark/container/
+!benchmark/container/**

.github/workflows/pages.yml

@@ -44,6 +44,7 @@ jobs:
           test -f _site/index.html
           test -f _site/styles.css
           test -f _site/app.js
+          test -f _site/benchmark-data.json
           test -f _site/robots.txt
           test -f _site/sitemap.xml
           test -f _site/assets/marginal-readme-hero.png

.github/workflows/swebench-lite-canary.yml

@@ -58,7 +58,12 @@ jobs:
           set -euo pipefail
           python -m pip install --upgrade pip
           python -m pip install -e ".[dev]"
-          python -m pip install modal "swebench[modal]"
+          python -m pip install "modal==1.5.3" "swebench==4.1.0"
+          python -m modal token set \
+            --token-id "${MODAL_TOKEN_ID}" \
+            --token-secret "${MODAL_TOKEN_SECRET}" \
+            --verify
+          test -s ~/.modal.toml
 
       - name: Validate evidence before Modal spend
         shell: bash
@@ -78,14 +83,15 @@ jobs:
           set -euo pipefail
           IDS="$(python benchmarks/swebench_lite/protocol.py ids --task-set "${TASK_SET}")"
           rm -rf evaluation_results
+          cp "${RUN_DIR}/baseline_predictions.ndjson" /tmp/baseline_predictions.jsonl
           RUN_ID="marginal-${TASK_SET}-baseline-${GITHUB_RUN_ID}"
           python -m swebench.harness.run_evaluation \
             --dataset_name princeton-nlp/SWE-bench_Lite \
             --split dev \
-            --predictions_path "${RUN_DIR}/baseline_predictions.ndjson" \
+            --predictions_path /tmp/baseline_predictions.jsonl \
             --instance_ids ${IDS} \
             --run_id "${RUN_ID}" \
-            --parallelism 10 \
+            --max_workers 10 \
             --modal true
           RESULT="$(find evaluation_results -type f -name instance_results.jsonl -print -quit 2>/dev/null || true)"
           if [ -n "${RESULT}" ]; then
@@ -102,14 +108,15 @@ jobs:
           set -euo pipefail
           IDS="$(python benchmarks/swebench_lite/protocol.py ids --task-set "${TASK_SET}")"
           rm -rf evaluation_results
+          cp "${RUN_DIR}/marginal_predictions.ndjson" /tmp/marginal_predictions.jsonl
           RUN_ID="marginal-${TASK_SET}-on-${GITHUB_RUN_ID}"
           python -m swebench.harness.run_evaluation \
             --dataset_name princeton-nlp/SWE-bench_Lite \
             --split dev \
-            --predictions_path "${RUN_DIR}/marginal_predictions.ndjson" \
+            --predictions_path /tmp/marginal_predictions.jsonl \
             --instance_ids ${IDS} \
             --run_id "${RUN_ID}" \
-            --parallelism 10 \
+            --max_workers 10 \
             --modal true
           RESULT="$(find evaluation_results -type f -name instance_results.jsonl -print -quit 2>/dev/null || true)"
           if [ -n "${RESULT}" ]; then
@@ -162,7 +169,7 @@ jobs:
 
       - name: Upload benchmark evidence
         if: always()
-        uses: actions/upload-artifact@v4
+        uses: actions/upload-artifact@v7
         with:
           name: swebench-lite-${{ inputs.task_set }}-${{ github.run_id }}
           if-no-files-found: warn

.gitignore

@@ -14,3 +14,10 @@ build/
 # MARGINAL local privacy keys
 *.privacy.key
 .marginal/
+
+# Local Codex benchmark trajectories, task checkouts, and copied runtime state
+/benchmark/runs/
+/benchmark/.venv/
+
+# Aggregated benchmark records are intentionally reviewable artifacts.
+!/benchmark/results/*.jsonl

README.md

@@ -27,6 +27,31 @@ Open source · Local first · Provider neutral · Zero mandatory runtime depende
 
 ---
 
+## First measured Codex integration — correctness first
+
+> **Exploratory 3-task smoke, one paired run per task.** This validates the integration; it is not a general performance claim.
+
+| Metric | Codex OFF | Codex + MARGINAL | Observed change |
+|---|---:|---:|---:|
+| SWE-bench resolved | 0/3 | 0/3 | **0/3 → 0/3** |
+| Effective tokens | 1,098,747 | 824,839 | **24.93% fewer** |
+| Effective latency | 593.11 s | 565.77 s | **4.61% lower** |
+| Tool calls | 33 | 32 | **3.03% fewer** |
+| Governance overhead | — | 0 tokens · $0 · **7.06 s** | measured separately |
+| Evaluator decision | — | **pass_through** | not eligible for a support claim |
+
+The matched trajectories used Codex CLI 0.147.0, GPT-5.6-sol, identical prompts and limits, and pinned SWE-bench Lite task images. The official verifier completed all six evaluations without infrastructure errors in the authoritative Docker run. Neither lane resolved a task, so tokens per resolved task is undefined: the observed 24.93% token difference is useful trajectory telemetry, not proof of useful token saving. **No deny was applied in these three agent trajectories**, so the difference cannot be attributed to a MARGINAL stop decision.
+
+The Modal audit completed two tasks per lane but hit the same image-build error for `pylint-dev__astroid-1978` in both lanes. Those cloud errors are preserved and excluded from scoring; the public evaluator now fails closed on `error_ids` and `incomplete_ids` instead of silently treating them as unresolved.
+
+[Full report](benchmarks/swebench_lite/PUBLIC_BENCHMARK.md) ·
+[Raw JSON](benchmarks/swebench_lite/public-benchmark.json) ·
+[Evidence bundle](benchmarks/swebench_lite/evidence/smoke-2026-08-11-dbce533/) ·
+[Frozen protocol](benchmarks/swebench_lite/README.md) ·
+[Modal workflow run](https://github.com/SignalLayerLabs/Marginal/actions/runs/31474500980)
+
+---
+
 ## The problem in one trace
 
 Coding agents can spend compute on actions whose incremental value is unclear or diminishing. The useful failure mode is not "GPT-5.6 is wasteful"; it is **repeated work against unchanged state that produces no new evidence**.
@@ -174,7 +199,7 @@ cd Marginal
 python -m pip install -e ".[dev]"
 ```
 
-The upcoming Codex reference integration remains a roadmap milestone. It is not presented here as already available.
+The auditable Codex reference adapter and its first matched smoke are now available in `benchmark/codex_adapter/`. Start from the frozen protocol and treat the current n=3 result as integration evidence, not a performance claim.
 
 ## Quickstart
 

benchmark/README.md

@@ -0,0 +1,77 @@
+# Codex baseline versus Codex + MARGINAL
+
+This directory contains the preregistration, runner, raw evidence, and analysis for a
+correctness-first paired evaluation of Codex with MARGINAL OFF and ON.
+
+No synthetic MARGINAL demo result is accepted as evidence for this comparison. Task
+correctness comes only from the benchmark's independent verifier. Failed, timed-out,
+empty-patch, and integration-failed runs remain in the dataset.
+
+## Frozen smoke set
+
+- Dataset: `princeton-nlp/SWE-bench_Lite`
+- Split: `dev`
+- Selection rule: the three lexicographically lowest SHA-256 digests of `instance_id`
+- Task-set digest: `1c8c06935484fd57f4c27c363418884542bcfb06c69d02f480b61fe7687b58eb`
+- Repetitions: one engineering run per condition
+- Conditions: `baseline` and `marginal`
+
+The smoke is an integration check, not publication-grade performance evidence. The
+20-task canary and larger repeated experiment remain gated on complete smoke telemetry.
+
+## Commands
+
+Run the non-verifier integration gate (this does not launch Codex inference):
+
+```bash
+.venv/bin/python -m benchmark.scripts.preflight --skip-verifier
+```
+
+Run the mandatory full gate before any benchmark trajectory:
+
+```bash
+.venv/bin/python -m benchmark.scripts.preflight
+```
+
+The full gate currently stops before inference for two scientific blockers:
+
+- Codex 0.147 does not expose shell exit status in `PostToolUse`, so the preregistered
+  successful shell-repeat mechanism cannot distinguish passing from failing commands; and
+- Codex is not yet running inside a pinned official per-instance SWE-bench environment.
+
+An official verifier backend is also required after those blockers are resolved. The task
+materializer and single-run adapter remain available for test-only integration work, but
+must not be used for benchmark inference on a bare host checkout.
+
+The future matched layout uses distinct worktrees for each lane:
+
+```bash
+.venv/bin/python -m benchmark.scripts.prepare_swebench_tasks \
+  --destination benchmark/runs/worktrees/baseline
+.venv/bin/python -m benchmark.scripts.prepare_swebench_tasks \
+  --destination benchmark/runs/worktrees/marginal
+.venv/bin/python -m benchmark.scripts.run_codex_task \
+  --instance-id pvlib__pvlib-python-1072 \
+  --condition baseline \
+  --worktree benchmark/runs/worktrees/baseline/pvlib__pvlib-python-1072 \
+  --run-dir benchmark/runs/pvlib__pvlib-python-1072/baseline
+```
+
+Run the MARGINAL condition against its separate checkout under
+`benchmark/runs/worktrees/marginal/`, then repeat for the other frozen instance IDs. The
+runner rejects a dirty tree, an attached branch, or a commit other than the frozen base.
+Do not begin the 20-task canary until all six smoke trajectories have complete raw
+telemetry and independent verifier results.
+
+The frozen design and implementation plan are under `docs/superpowers/`. The full
+preflight remains the authority: do not launch a trajectory unless it passes.
+
+## Safety
+
+- Do not place API keys, Codex authentication files, prompts containing secrets, source
+  content, or raw tool output in tracked results.
+- The runner isolates `HOME`, gives tool subprocesses an allowlisted environment, keeps
+  `CODEX_HOME` out of that environment, and fails closed if exact authentication material
+  appears in any changed file or extracted patch.
+- Local runtime data belongs under `benchmark/runs/` and is ignored.
+- Do not commit or push benchmark work unless explicitly authorized.

benchmark/__init__.py

@@ -0,0 +1 @@
+"""Reproducible Codex benchmark integration for MARGINAL."""

benchmark/analysis/failure_analysis.md

@@ -0,0 +1,17 @@
+# Smoke Failure Analysis
+
+No trajectories are available yet. Observations and interpretations will be kept separate.
+
+## Preflight blocker
+
+- Observation: Codex 0.147 `PostToolUse` exposes shell output but not exit status. Treating
+  every completed shell handler as successful would corrupt MARGINAL's successful-repeat
+  history; treating every shell as failed makes the preregistered primary intervention
+  effectively pass-through for searches and tests.
+- Observation: task checkouts are materialized at the correct commits, but Codex is not
+  yet executed inside the official pinned per-instance SWE-bench runtime.
+- Observation: no official verifier backend is available on this host.
+- Interpretation: executing trajectories now would measure an unfaithful mechanism in an
+  uncontrolled task environment and would spend tokens without admissible correctness.
+- Resolution: introduce a tested success-observable interception boundary, run Codex in
+  the official task environment, configure the official verifier, and rerun full preflight.

benchmark/analysis/summary.md

@@ -0,0 +1,81 @@
+# Codex OFF versus Codex + MARGINAL: Smoke Report
+
+## Executive Summary
+
+No Codex OFF/ON trajectory was executed. The integration-only preflight passes for the
+frozen three-task set, Codex 0.147.0, `gpt-5.6-sol`, and `high` reasoning effort. The
+mandatory full preflight stops before inference because successful shell executions are
+not observable through the pinned hook contract and no official per-instance SWE-bench
+execution backend is implemented. An official verifier is also unavailable on this host.
+
+Running anyway would spend tokens on an experiment that cannot faithfully test the frozen
+mechanism or produce admissible correctness. There are therefore no performance claims.
+
+## What MARGINAL Actually Does
+
+MARGINAL is a provider-neutral action-governance library built around a `Treasury`, policy,
+budgets, decision ledger, and optional state-aware diminishing-return detector. It does not
+natively modify Codex prompts or model calls. The benchmark adapter maps official Codex
+tool hooks to MARGINAL actions and can deny a proposed action in enforce mode. The baseline
+has no MARGINAL process, hook, state, or context.
+
+## Methodology
+
+The preregistered smoke uses three deterministically selected SWE-bench Lite `dev` tasks,
+one run per condition, separate exact-base-commit checkouts, identical frozen prompt/model/
+reasoning/sandbox/timeout settings, raw JSONL telemetry, and the official SWE-bench
+resolution criterion. Correctness is primary. The smoke is an integration check and does
+not support population-level performance inference.
+
+## Correctness Results
+
+| Metric | Codex baseline | Codex + MARGINAL | Delta |
+|---|---:|---:|---:|
+| Executed tasks | 0 | 0 | 0 |
+| Resolved tasks | unavailable | unavailable | unavailable |
+| Resolution rate | unavailable | unavailable | unavailable |
+
+## Paired Results
+
+- `BOTH_SOLVE`: unavailable
+- `BASELINE_ONLY`: unavailable
+- `MARGINAL_ONLY`: unavailable
+- `NEITHER`: unavailable
+
+## Efficiency
+
+Tokens, time, actions, tool calls, repeated behavior, files modified, and diff size are all
+unavailable because no comparative trajectory was launched.
+
+## Intervention Analysis
+
+No benchmark intervention occurred. Beneficial, neutral, harmful, indeterminate, and
+false-positive rates are unavailable.
+
+## Regressions
+
+No regression was observed because no paired outcome exists. This is not evidence of no
+regressions.
+
+## Failure Analysis
+
+The blocking observations and their interpretation are recorded in `failure_analysis.md`.
+They are infrastructure and measurement failures, not model/task outcomes.
+
+## Limitations
+
+- Codex 0.147 unified-shell `PostToolUse` omits exit status, preventing defensible
+  successful-repeat history for the main search/test/debugging surface.
+- Codex is not provisioned inside an official pinned per-instance SWE-bench environment.
+- No official verifier backend is available locally.
+- The frozen set contains only three tasks and one repetition even after feasibility is
+  restored; any future smoke result will remain exploratory.
+- Agent and environment nondeterminism require larger repeated paired runs for credible
+  generalization.
+
+## Conclusion
+
+The evidence is insufficient to answer whether MARGINAL improves task completion, reduces
+unproductive behavior or cost, intervenes incorrectly, helps or hurts particular task
+classes, or generalizes beyond stress tests. The current evidence is not strong enough to
+justify enabling MARGINAL with Codex on the basis of this benchmark.

benchmark/codex_adapter/__init__.py

@@ -0,0 +1 @@
+"""Codex hook adapter components used by the benchmark harness."""