Review-effort dials: size the review machinery to the task, with postmortem observability

[parsakhaz]

The observation (Parsa, 2026-07-13)

When we orchestrated this loop by hand, review effort was a set of knobs we turned up or down per task. In the current pipeline those knobs are welded to maximum for every run, whatever the task size. That compounds fast in both cost and wall-clock.

The dials

High-level (per run):

1. Dual-lane vs single reviewer — do both Codex and Claude review this, or is one lane enough?
2. Review-loop pass count — how many find→fix→re-review rounds each loop gets (plan review and post-PR review).
3. Frontend QA verifier — the computer-use verifier driving the running app; expensive, not always warranted.
4. End QA pass — the PR-test-automation-style manual-tests execution at the very end.

Low-level:
5. Per-dispatch reasoning effort — what effort tier each sub-agent session runs at.

Current state

The light/full lane call in /do Step 1 is the only dial that exists today, and it only modulates the pass caps (1 vs 3) and the research dossier. Dials 1, 3, 4, and 5 are fixed at maximum for every run.

Ask (near-term, not a redesign)

Observability first. The postmortem already reviews how a run ran (wall-clock, active vs idle). Extend it to record, per run:

• what the dial settings effectively were (lane, passes used per loop, which verifiers/QA ran, effort tiers)
• the yield: findings per review pass per lane (did the second Codex pass catch anything the first didn't? did the Claude lane catch anything Codex missed, and vice versa?), QA-pass findings vs its cost
• a judgment line: overdone / right-sized / underdone, with the one dial that would have changed it

That gives us tuning data while we run. Eventually this becomes evals; for now the postmortem is the measurement instrument, and the wrap-up can carry the dial summary so the human sees cost-shape at the PR.

Later (once data exists): the lane call grows from light/full into a dial profile — e.g. docs-only changes get single-lane review and no QA pass; auth/schema changes get everything.

Explicitly out of scope for now: changing any default. Keep it top of mind, measure, then tune.

🤖 Captured with Claude Code from Parsa's dictated notes

[parsakhaz]

Proposed mechanism for building the intuition (discussion with Parsa, 2026-07-13)

Four stages, each building on what exists:

1. Structured records — the postmortem writes a machine-readable dial block (lane, passes used, findings per pass per lane, verifiers/QA run + findings, wall-clock). Postmortems-as-labeled-issues makes this a queryable corpus.
2. Scheduled aggregation — a weekly loop sweeps postmortem issues and emits the dial report (yield per dial across runs). This is where priors become explicit.
3. Profile table + bounded deviation — stable findings become change-type × dial rows, changed only via the postmortem one-change mechanism. The lane call picks a profile; the model may deviate one notch with a stated, recorded reason. Deviations are themselves data.
4. Retrospective evals for free — per-lane-per-pass findings hold the counterfactuals ("would single-lane have shipped what dual caught?"). Real eval harness later; the corpus starts now.

Trust policy for the LLM:

• High trust on classification (surfaces touched, blast radius, user-visibility) — it produces the inputs to the dial decision.
• No trust on de-escalation by feel: turning a dial down removes the mechanism that would catch the error of turning it down — skipped reviews fail silently, extra reviews fail visibly on the invoice. Dial-downs come from the table's measured yield only; dial-ups are always allowed.
• Plan-time decision from the item, never by the producing session (same fresh-context separation as the Rules).
• Hard floors as rules, not data: production-adjacent surfaces (auth/schema/money/migrations) never drop below dual-lane + QA, mirroring the red tier.
• Deviation budget widens only when the aggregate shows the model's deviations beat the table.