policy: rank levers from what previous runs measured

[Gaurang-2005]

What this closes

#118 gave the loop the ability to read its own past. Nothing called it. select_interventions still scored every candidate as coverage × expected_delta_mean — a hand-authored constant, identical on every run no matter what the last one measured — so a lever that lost 9% last night ranked exactly where it did before anyone had ever run it.

This wires the record into the ranking, and asks once whether to use it. It is the return edge that makes the thing a loop rather than a pipeline:

run N   Phase 5  write_verification ──┐
                                      │
run N+1 Phase 3  load_history  ◄──────┘

Substitution, not a blend

expected_delta_mean is an estimate of a lever's effect. A delta that lever actually recorded on this GPU is a better estimate of the same quantity, so it replaces it. Coverage — a property of the current trace — is untouched.

There is deliberately no weighting constant between prior and measurement. A number like K = 2 reads as calibrated and is not; playbook/match.py refuses to pick such a threshold and says so, and this follows it. For the same reason ranking stays a precedence tuple rather than one blended score — terms answering different questions should not collapse into a scalar where one can quietly outvote another:

rejected by gate → estimate ≤ 0 → demoted → magnitude → name

"Not worth a run" sits above the demotion, and finding that out is what the end-to-end check was for. With the demotion higher, a lever measured at −9% in every run outranked a conflicted lever estimated at +1.4%, because the loser's record did not disagree with itself. That spends the run re-confirming a result already in hand. Estimate sign first, then consistency among the levers that might actually help. Unit tests alone did not show this; driving it against a real runs/ directory did.

What it looks like

Three levers with identical coverage, against a real history:

	--no-history	--use-history
enable_expert_parallel	+0.0167	+0.1550 measured
kv_cache_dtype_fp8	+0.0167	+0.0144 measured, demoted
enforce_eager	+0.0167	−0.0300 measured

Off, it is a three-way tie broken alphabetically — which is the problem. On, they separate by evidence.

Four rules

• A conflicted record demotes but never removes. Won twice and lost twice is not a measured neutral: the lever behaved differently under conditions the record does not capture, which makes it the weaker bet while that holds. The demotion lifts by itself once the record stops disagreeing — it describes the evidence, not the lever.
• A known loser still ranks last, below an uncertain candidate.
• No GPU SKU means no substitution, not a guess. A result measured on an H100 says nothing about an MI355X — the single mistake the record's GPU key exists to prevent.
• History changes what gets tried, never what gets kept. The rollback gate still decides, and every candidate still wins or loses its own A/B on this run.

The question, and where it lives

gitm run asks once, before any capture, when previous runs left results and there is a terminal to ask at:

5 previous run(s) left measured results.
  [Y] rank this run from them   [n] ignore them and score from the catalog
  Nothing is deleted either way. No answer within 60s uses them.
>

Nothing can be overwritten. Each run writes into its own runs/<uuid4>/, so a new run never lands on an existing verification.json. The choice is only whether this run reads the earlier exports; declining leaves every one of them on disk. Measurements that cost GPU time are not discarded by a keystroke.

No answer means yes. An unattended run must not sit on a prompt, and of the two answers using the record is the one that throws nothing away. Without a terminal there is nobody to ask, so it takes the same default at once rather than waiting out the timeout against a pipe that will never reply.

The prompt is in the CLI, not the loop — 17c619c moves it there. gitm.optimize is the embedded entry point a vLLM process calls in-line, and a function that can block on stdin is one such a caller cannot use. Nothing would have appeared on a terminal anyone was watching; the run would simply have stopped for sixty seconds inside someone else's process. --use-history / --no-history decide it without a prompt for scripted runs, and an explicit flag is never re-litigated.

Behaviour change on merge, stated plainly

For gitm run, the effective default becomes on, since silence means yes. For optimize() / run_loop(), it stays off unless explicitly passed — an embedded caller's ranking does not change because of an upgrade.

Not in this PR

• Nothing here reads the deviation table. deviation: a typed table, ranked by recoverable time #119's deviation_table.py still has no consumer. This PR shares no file with it, imports nothing from it, and branches from the same commit — the two are independent and can merge in either order.
• largest_residual is untouched; autoresearch still ranks by mean fractional overshoot.

Validation

• 1276 passed, 1 skipped, exit 0; ruff clean across gitm/ and tests/.
• 19 new tests. Every behavioural one was confirmed to fail against the code it fixes, not merely to pass against the fixed version.
• Two are structural rather than behavioural: one reads the scheduler's source and fails if select.select or a prompt helper ever drifts back into it; the other pins that optimize takes the answer as a parameter and contains no input(. Both guard the boundary rather than today's arrangement of it.
• Also driven end to end against a real runs/ directory from generate_demo_runs.py (merged in history: read previous runs' measured results back #118) — that is where the precedence bug above surfaced.
• Prompt tests drive a real os.pipe(), not a StringIO: the prompt waits on select, and a stream that cannot be selected would be testing something the run never does.
• No run against a live GPU.

🤖 Generated with Claude Code

Confidence Score: 4/5

The PR is not yet safe to merge because malformed historical metadata can still abort a run when history is enabled.

Findings

1.  Malformed history can abort runs ▶

Summary

This PR makes historical optimization results available to intervention ranking while preserving current-trace coverage and isolating measurements by GPU and workload fingerprint.

• Adds explicit CLI controls and an interactive default for history-backed ranking.
• Keeps embedded API and scheduler entry points non-interactive and history-disabled unless requested.
• Substitutes validated historical deltas for catalog estimates and demotes conflicted records.
• Filters history by GPU SKU and workload fingerprint.
• Rejects non-numeric and non-finite historical deltas.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Previous verification exports] --> B[Load history]
  B --> C{GPU SKU and fingerprint match?}
  C -- No --> D[Ignore record]
  C -- Yes --> E[Validate finite delta]
  E --> F[Rank interventions]
  G[Current trace coverage] --> F
  H[Catalog estimate] --> F
  F --> I[Selected experiments]
  I --> J[New verification export]
  J --> A

Loading

_{Reviews (3) · Last reviewed commit: "history: the isinstance form ruff 0.12 a..."}

[greptile-apps]

Malformed history can abort runs

When history is enabled, this makes every prior export part of the run-critical Phase 3 path, but the reader only verifies that each result is a dictionary. A malformed but parseable export can raise while computing speedup - 1.0, or pass a non-finite delta into ranking and JSON output. Because this happens after capture, one damaged old export can abort or corrupt an otherwise expensive run; validate and skip malformed result fields or isolate failures per export.