AUTOQUANT

Which coding agent is the better quant researcher?

Quick start · Protocol · Score · Dashboard · How it works · Agent program

---

Five coding-agent configurations start from the same seed strategy, edit one file, and race the same cost-aware walk-forward score. Git keeps validated wins. results.tsv counts every attempt. A protected evaluator owns data, timing, costs, gates, and scores.

This is a research benchmark — no broker, testnet, wallet, or live path.

Inspired by karpathy/autoresearch: one mutable genome, one frozen metric, overnight autonomy.

Protocol

┌─────────────┐     edit only      ┌──────────────┐     score.json     ┌────────────┐
│  harnesses  │ ───────────────▶   │  strategy.py │ ───────────────▶   │ evaluate.py│
│  ×5 agents  │    50 experiments  │   (genome)   │   protected I/O    │  (oracle)  │
└─────────────┘                    └──────────────┘                    └────────────┘
        │                                                                 │
        │  worktrees · budgets · audit · one-shot lockbox                 │
        └────────────────────── arena.py ◀────────────────────────────────┘

Artifact	Role
strategy.py	Mutable genome — the only file competitors may edit
evaluate.py	Trust boundary — splits, causality, t+1 fills, costs, gates, score
run.py	Hash verifier + launcher
program.md	Full experiment loop for the harness
arena.py	Worktrees, supervision, budgets, audit, lockbox
dashboard.py	Self-contained race report → docs/index.html

Score

score = mean(fold Sharpe) − 0.5 × std(fold Sharpe)

Metric	Six non-overlapping 4-month OOS folds on VALID
Fill	Signal at close t → fill at open t+1
Costs	5bp taker · 1.5/5bp half-spread · √impact · funding
Gates	Causality · DD ≤ 25% · turnover ≤ 40× · ≥20 trades · no raw leverage
Search ratchet	Working parent: Δscore > materiality (0.02), ≥4/6 folds, cost-aware
Official ratchet	Champion: Δscore > noise floor (+ attempt correction), ≥5/6 folds
Archive	Four lineages: best · stable · cost-robust · distinct
Budget	Exactly 50 logged experiments per competitor

TRAIN     2019-09 → 2022-12     hypothesis formation
VALID     2023-01 → 2025-06     six OOS folds  (race metric)
LOCKBOX   2025-07 → 2026-06     one-shot open after all runs terminal

Competitors

Roster is defined in competitors.toml (what the arena actually runs):

Agent	Harness	Model	Color
grok-4.5-high	Grok	grok-4.5 (high)	#1D9BF0
codex-luna-high	Codex	gpt-5.6-luna (high)	#A78BFA
codex-terra-high	Codex	gpt-5.6-terra (high)	#10A37F
codex-sol-high	Codex	gpt-5.6-sol (high)	#F59E0B
gemini-3.6-flash-high	Antigravity CLI	gemini-3.6-flash-high	#4285F4

Same seed · same data · same evaluator · same kickoff · no human intervention after start.

The original three-way race completed 450 experiments and opened its one-shot lockbox before Gemini 3.6 Flash and Codex Sol were added. Post-lockbox extensions keep that original state and every worktree unchanged, start from the same original baseline, and are comparable on VALID only. They cannot receive retroactive lockbox scores without violating the published one-shot protocol.

Results snapshot

Agent	Attempts	Retained VALID champion	Keeps	Scope
grok-4.5-high	150	1.819915	3	original race + lockbox
codex-sol-high	50	1.511608	2	post-lockbox VALID only
gemini-3.6-flash-high	150	1.199804	2	post-lockbox VALID only
codex-terra-high	150	1.065688	1	original race + lockbox
codex-luna-high	150	0.916789	1	original race + lockbox

Sol's 50-attempt result used gpt-5.6-sol with high reasoning and produced two accepted ratchet steps: seed 0.483266 → experiment 1 0.884258 → experiment 34 1.511608. All 50 rows reconcile to evaluator-authored score artifacts and the protected-file audit is clean. Attempt budgets differ across the historical snapshot, so it is not a controlled same-budget model ranking. Post-lockbox extensions have no sealed-period result.

Quick start

Python 3.11+ and uv.

uv sync
uv run python setup_data.py                 # pin Binance USD-M public archives
uv run python evaluate.py --calibrate       # baseline + noise floor
uv run python arena.py doctor
uv run python arena.py start --competitors all --max-experiments 50
uv run python arena.py status
uv run python arena.py lockbox              # once, after every run is terminal
uv run python arena.py report --out docs/index.html

To append a missing competitor with the current 50-experiment budget:

uv run python arena.py start --competitors codex-sol-high --max-experiments 50

Extensions refuse duplicate competitors, preserve the original baseline, record their own explicit attempt budget, and merge new state into the existing terminal records rather than replacing them.

Smoke test without market history:

uv run python setup_data.py --synthetic --force
uv run python evaluate.py --calibrate --samples 40
uv run python arena.py doctor --skip-harnesses
uv run python run.py
uv run python dashboard.py --out docs/index.html

--synthetic is explicit and never a silent fallback. Real arenas must reject synthetic data.

Protocol v2 (two ratchets)

v1’s single ratchet used a coarse seed-perturbation “noise floor” (~0.24) as the only step size, discarded every smaller improvement, and reset the branch — so most recursive progress never appeared on the race curve.

candidate
   ↓
search acceptance → lineage archive → next mutations
   ↓
official promotion → confirmed champion
   ↓ once
sealed lockbox evaluation

Layer	Status	Rule of thumb
Working parent	search	Material multi-fold gain vs parent; enables recursion
Official champion	keep	Strong noise-floor bar; confirmed promotion
Archive tip	archive	Improves stability / cost / diversity slot only

Each score.json includes structured comparison feedback (score/fold/cost/drawdown deltas, archive correlations, gate headroom) and a decision block agents must follow. Recursion metrics (confirmed_promotions, AUC of working vs official curves, attempts per keep) live in recursion_metrics.json and the lockbox summary.

Design choices

• One file to modify. Agents only touch strategy.py. Diffs stay reviewable.
• Fixed experiment budget. Fairness is 50 attempts, not wall clock.
• Two ratchets. Search accepts stepping stones; official keeps stay hard to earn.
• Tamper-evident, not a hostile sandbox. Hash checks, import gates, isolated I/O, protected diffs, and full trial accounting make cheating visible. They do not stop a malicious local process with the user's filesystem authority.
• Multiple-testing aware. The dashboard estimates effective N, False Strategy thresholds, Deflated Sharpe, and PBO when enough trusted trials exist. Every journal row is counted; only evaluator-authored score.json rows enter statistics.
• Research only. The lockbox is a consistency check, not a live season. Frontier models may have memorized recent crypto history.

Dashboard

Eight panels, one HTML file (Chart.js CDN only): dual-ratchet race (solid official / dashed working) · every attempt · hit rate · lockbox equity · discovery timeline · corrected leaderboard with recursion metrics · champion correlation · 30s replay.

uv run python dashboard.py --out docs/index.html
uv run python dashboard.py --watch          # regenerate every 60s

Open docs/index.html after clone. Companion explainer: docs/how-autoquant-works.html.

What is not claimed

Repository tests use a deterministic fixture for accounting and control-flow invariants. They do not establish crypto alpha. Never label a synthetic score as a market result. Publish a leaderboard only after a real download, verified manifest, full calibration, arena doctor, and a completed arena run.

---

Data honesty notes

• Binance USD-M monthly kline archives 404 before 2020-01; TRAIN starts 2019-09 by protocol, first observation is 2020-01.
• Universe freezes 20 assets liquid on Binance spot at train start that later obtained perps — including failures (LUNA, FTT, SRM). Not 20 contemporaneous perps on day one.
• Final ~54 days of VALID are intentionally unscored rather than a short seventh fold.
• See evaluate.py, setup_data.py, and program.md for the full contract.