epic: token-consumption benchmark for the compression refactor

[thewrz]

This was written agentically; verify its assertions:

Design: docs/superpowers/specs/2026-08-13-token-benchmark-design.md (branch docs/token-benchmark-spec)

Why

The compression sprint (#104–#108, PRs #115→#120) cut the skills down hard, and we have no evidence of what that bought at runtime. The claim is easy to overclaim, so this epic splits it into two questions with different standards of proof.

Q1 — did the compression happen? A static property of three files at two commits. Deterministic, free, already answered:

Surface	06d18cf	53e7e8c	Δ
Resident (always-loaded SKILL.md)	~69,579 tok	~28,242 tok	−59.4%
Reachable (+ references/ + .shared/)	~69,701 tok	~63,492 tok	−8.9%

That gap is the finding. ~41k tokens left the always-loaded spine, but only ~6k left the corpus — the other ~35k moved into 9 references/ files that did not exist at baseline. So the saving is conditional on a behavior, not a byte count:

• references rarely read → ~41k saved per session
• references usually read → ~6k saved, plus added file-read round-trips and tool calls, which can net out worse

Q2 — is the plugin cheaper to run end-to-end? Stochastic. Published work on agentic coding tasks reports per-task token usage varying up to 30× across repeated runs of the identical task; the minimal-pair cleanliness study needed 660 trials to resolve a 7–8% effect. One run per arm adjudicates nothing.

What

A two-tier benchmark. Tier 0 is the verdict; Tier 1 is corroboration with error bars.

Tier 0 — static accounting of resident vs reachable surface at both arm SHAs, using the same bytes/4 estimator tests/lint-skill-size.sh already uses so the two cannot disagree.

Tier 1 — containerised A/B on a purpose-built mock repo:

• Arms: old = 06d18cf (pre-sprint), new = 53e7e8c (post-wave). Both accept the benchmark command verbatim — --yolo, --fast-mode, --auto-review, --auto-serialize all exist at both SHAs, and Max 10 issues / chain depth cap: 4 are byte-identical strings.
• Trial: one fresh Debian 13 container, one fresh repo, one command: /parallel-issues --yolo --fast-mode --auto-review --auto-serialize with all ten issues in Ready. The skill selects, orders, and chains them itself.
• Mock repo: "Tally", a small ES-module SPA. Ten spec-grade issues — exactly five conflicting (two collide in render.js, three in store.js) and five disjoint, with native blocked_by edges aimed at chain tails to keep the longest chain at depth 3, one under the cap.
• Oracle: a hidden zero-dependency node --test suite injected after the run. An issue scores only at 100% of its cases, so a run that spends fewer tokens by doing less work loses rather than wins.
• Primary instrument: reference hit rate, not token total — it is the mechanism that decides whether Tier 0's paper win is real.
• Effort sweep: feat(skills): make worker model and reasoning effort configurable via .agent/config.env #125 landed, so AGENT_WORKER_EFFORT sweeps low/medium/high from .agent/config.env on the new arm. The highest-value comparison is new@low vs old@high: effort tiers move tokens by multiples while the refactor moves them by percents, so a dominance result survives the noise floor that a percentage delta does not.

Longitudinal operation

This is standing infrastructure, not a one-shot study. An arm is any plugin SHA; 06d18cf and 53e7e8c are the first two points. Results accumulate into a series showing how the project fares across releases, models, and effort tiers.

• Results ledger — bench/results/*.jsonl, append-only and committed, keyed by (plugin_sha, fixture_version, model, effort). Never rewritten, so every chart is a pure function of the ledger and can never disagree with its source.
• Fixture freezing — a series is meaningful only while the task holds still. Improving an issue body or repairing a flaky oracle case silently makes every earlier point incomparable, and the graph bends rather than warns. So any change under bench/fixtures/, bench/issues/, or bench/accept/ forks fixture_version, and points from different versions are never plotted on one line.
• Drift control arm — gpt-5.6-luna in six months is not gpt-5.6-luna today. Provider-side change moves the numbers with no plugin change at all, and an uncontrolled series cannot distinguish "agent-kit regressed" from "the model changed underneath us." Every round re-runs the frozen 06d18cf alongside whatever is being measured; the delta against its own earlier result is the correction every other point is normalised against. A round without one is excluded from trend claims.
• Cadence — Tier 0 on every merge to main (it is a script over the repo, so it is free; lint-skill-size.sh already computes those numbers to gate on, and a CI step records rather than discards them). Tier 1 stays rare. Dense free trace, sparse expensive anchors, same axis.
• Sampling, not a grid — models × efforts × SHAs is not affordable as a full matrix. Default round is the drift control plus current main at each effort tier. A new model enters by running the frozen arm on it first, establishing that model's own baseline, before any agent-kit claim is made on it.

Staging

Spend is gated; per-trial cost is unknown until measured.

• Tier 0 — bench/tier0.sh, any SHA, appends to the ledger
• Ledger — bench/results/*.jsonl schema, append-only, fixture_version from day one
• Pre-registration — bench/PREREGISTRATION.md committed before trial 1
• Fixtures — Tally skeleton, gold tree, ten issue bodies, hand-authored .agent/config.env (frozen set)
• Oracle — bench/accept/ + vendored DOM stub (frozen set)
• Harness — container image, repo reset, invocation, parse-rollout.py
• Tier 0 in CI — record resident/reachable on every merge to main
• Stage 1: pilot (2 trials) — validate harness, learn real cost/wall-clock/GraphQL per trial → go/no-go
• Stage 2: dominance probe (5 paired trials) — new@low vs old@high → go/no-go
• Stage 3: frontier (2 arms × 3 tiers) — acceptance score vs blended USD, two curves

Known threats

• The arms differ by more than compression — ~20 merged issues separate them, including fix(chains): stacked PRs inherit green checks and approvals earned against a base that no longer exists #122's stacked-PR retarget fix (present on new, absent on old). Correctness fixes cut both ways. Tier 0 is the compression claim; Tier 1 is a whole-plugin claim, and the spec says so rather than blurring them.
• Concurrency cap is a first-order confounder — both arms read max_concurrent_threads_per_session from ~/.codex/config.toml and are contracted to stop and ask when it is absent. It must be present and identical in both containers, or parallelism differs silently, or the unattended run deadlocks by design.
• Memory must be off — it is stateful across trials, so trial N would contaminate trial N+1.
• Board work burns GraphQL on the same account as ordinary work; the harness gates on gh api rate_limit and pauses rather than failing mid-run.

🤖 Co-authored by Claude Opus 5.

---

2026-09-14 scope and acceptance update

This was written agentically; verify its assertions:

Add correctness-conditioned orchestration efficiency to the standing benchmark

Keep the existing Tier 0/Tier 1 separation, oracle, frozen fixtures, drift controls and pre-registration. The peer proposal to measure successful issue completions per 100 model turns fits this epic, but only if “completion” is independently acceptance-verified and includes the configured workflow obligations.

Added acceptance

• Add accepted issue completions per 100 model turns alongside accepted completion rate, total/per-completion tokens and cost, wall time and unresolved/parked work. Report numerator, denominator and terminal criterion (draft/receipt/CI versus merge when authorized).
• Include incomplete/failed runs rather than dropping them to make efficiency look better; distinguish observation cutoff from terminal success. Zero completions and censored timing have explicit handling.
• Consume test(bench): measure interface-discovery ops — ~23 per run spent re-deriving documented helper interfaces #710's versioned root/leaf metrics, including duplicate actions, full-suite executions, time/tokens to first edit and overhead after verified correctness where measurable.
• Record model/effort, loaded kit identity, harness capabilities, fixture version and declared root/leaf topology. Use matched repeated trials and existing drift controls; do not infer Luna-vs-Terra superiority from this single watched run.
• Preserve hidden-oracle correctness and all required review/verification obligations so fewer turns cannot win by skipping work.
• Amend measurement schema/pre-registration prospectively; retain original trial records and explicitly mark unavailable historical metrics rather than rewriting them.
• Include deterministic adversarial fixtures for partial dispatch, stale/mixed handback, delayed review completion and skipped acceptance, separate from paid live trials.

The P0 issues (#722, #717, #726, #727, #728) define immediate reliability invariants. Benchmark work must not delay those fixes or claim their observed failures have already been reproduced under a correctly activated plugin. Compact worker contracts, failure routing, verification reuse and hook rewriting should earn broader rollout through these measurements.

🤖 Co-authored by Codex.