Review outer loop gen-0 auto-merged commits

[lukeinglis]

Context

Outer loop generation 0 ran with the corrected forecastbench benchmark config (#161, PR #162). 12 candidate topologies were evaluated — all successful candidates hit 7/11 gate rungs (up from 5/11 baseline). Five commits were auto-merged to main by sub-CEOs. These need human review before we trust them.

Commits to review (oldest first)

1. 0f4701f — Enable extended thinking for Claude forecaster models

• Adds THINKING_BUDGET = 10000 tokens for Claude models
• Omits temperature when thinking is active (API requirement)
• Review: Does this actually improve calibration, or just add latency and cost? The candidate that made this change timed out on the gate (scored 0.0).

2. 4a8a1a5 — Add market probability anchoring to improve Brier Index

• Blends forecasts toward market price with weight 0.91 via _apply_calibration in eval.py
• Claims Brier Index 61.285 → 62.457 (+1.172 points)
• Review: This is the most impactful change. Is the 0.91 weight overfitting to the 2 pinned rounds? Does _apply_calibration belong in eval.py or lab_forecaster.py? Is this legal under baseline track rules (it's post-processing, not search/RAG)?

3. a5f1f33 — Add structured logging across codebase

• Coverage 18% → 54% (128/238 functions)
• Review: This is NOT a Brier improvement — it's a backlog leak from the trimmed backlog. The sub-CEO picked up the observability item despite trimming. Consider reverting if it's noisy, or keeping if the logging is useful.

4. 2e28d58 — Increase thinking budget and add extremity clamping

• Thinking budget 10000 → 12000
• Adds extremity clamping [0.02, 0.98] in _apply_calibration
• Claims Brier Index 62.457 → 62.489
• Review: Builds on commits 1 and 2. The 0.032 improvement is tiny — is this noise?

5. 512256e — Tune calibration parameters and thinking budget

• Market anchor weight 0.91 → 0.94
• Extremity clamping 0.02/0.98 → 0.04/0.96
• Adds dataset shrinkage (6% toward 0.5)
• Thinking budget 12000 → 16000
• Review: Stacks more tuning on top of commits 2 and 4. Are these parameters overfitting to the 2 pinned rounds? The dataset shrinkage is a new mechanism — does it make sense conceptually?

Key questions

1. Are the calibration parameters overfitting? All tuning was done against 2 pinned rounds. We should test on held-out rounds before trusting these values.
2. Is market anchoring legal for baseline track? Post-forecast blending toward market prices could be considered "tools" under competition rules.
3. Should we revert the logging commit? It's off-topic and adds noise to the diff.
4. Should we update the gate baseline? If the improvements are real, re-center the ladder with python gate/make_manifest.py --set-baseline.
5. Do the stacked parameter changes compose well? Each sub-CEO tuned independently — the combined effect may not be the sum of parts.

Outer loop observations

• Total cost: ~$206 across 12 candidates
• Topology signal was confounded by auto-merge — all candidates hit 7/11 because earlier candidates' improvements accumulated on main
• 2-node topologies scored highest, but only because of parsimony penalty (all had same rung count)

[lukeinglis]

First step: measure the current Brier Index

Before reviewing individual commits, run the gate to see where the score actually landed after all 5 commits:

# Dry run (dummy forecaster, no API cost) to verify gate plumbing:
uv run python gate/make_manifest.py --dry-run

# Real score (lab forecaster, uses API):
uv run python gate/make_manifest.py --set-baseline

The outer loop reported 7/11 rungs passing, which means the Brier Index is somewhere between 62.285 and 63.285 (baseline 61.285 + 1 to +2). The exact number tells us how much headroom we have and whether the stacked parameter changes actually compose well.

Only update the baseline if the score is real and we're keeping the changes. The --set-baseline flag writes gate_baseline.json, which re-centers the ladder.