PLAN: evaluation rubric v2 — enforce couplings, verdict vocabulary, instrument fixes (from the three-way design review)

[mmcky]

Orientation, added 2026-08-07 — read this before picking anything up here. The product was reframed on 2026-08-07: triage — should this lecture be converted at all? — is now the skill's front door, and review mode is kept for when a conversion PR exists (#33, benchmark 0.4.0). Nothing on this issue was invalidated by that: #33 deliberately left the rubric, weights, gates, bands and scorecard JSON untouched, and both anchors reproduce byte-identically. But the five open items below were written when review mode was the product, so weigh them accordingly — the readability instrument in particular sits in the layer that got demoted, and the evidence for demoting it is that in every evaluation to date the recommendation was decided by the triage-layer instruments and never moved by the scorecard on top. This issue has not moved since 2026-07-27 and is the one genuinely untouched thread in the benchmark programme.

Two independent design critiques of the benchmark evaluation system — a fresh unframed review session and a 36-agent adversarial workflow in which every critique faced a steelman defense of the original design — are merged in reviews/benchmark-design-2026-07-21-merged.md (independent report alongside it). This issue tracks the v2 changes that survived the defense. The measurement foundation (as-used doctrine, evidence→engine determinism, verbatim extraction) was independently endorsed by both reviews and is not in scope — this is a scoring-superstructure revision.

Corrections of record already applied on the PR #6 branch (findings briefly posted to lecture-python.myst#654 were withdrawn — that PR gets one authoritative evaluation after v2): the markov_asset lecture does execute in notebook order — a stale global masks the stray err.throw(), silently disabling the checkify stability validation (subtler than the "does not build" we claimed); both reference replays deviate from their lectures' construction patterns; the as-used totals are single passes, not medians.

Engine changes (validatable against the committed evidence files)

• Enforce the safety couplings in code — derive the logic-design bug cap from builds/x64-divergence inside score_all; verdict gates: correctness 1 caps at "net regression", correctness 2 caps at "mixed/wash". Closes every demonstrated hole incl. the honest-evidence 4.2 (float32 catastrophe, no logic bug → "clear improvement — merge")
• "No-conversion" verdict output — reconciles review and triage: a polished ge_arrow currently scores 4.0 "merge" while triage says don't-convert at a 0.028 s baseline
• Sensitivity stamp in score.py — perturb each boolean and band-adjacent value; mark scorecards robust/fragile with deciding flips listed (one concept currently flips markov_asset's band; one boolean flips ge_arrow's)
• K-repeat as-used measurement — median of ≥3 fresh-process runs, contested-band annotation when the spread crosses an edge

Documentation honesty pass

• Thresholds labeled as policy choices with derivation notes ("calibrated" → "anchored"; the wash band's noise-floor rationale recorded)
• Fan-out paragraph: a root-cause fact's total influence is the sum over its manifestations — currently unstated and the weight table invites misreading; dedupe the near-verbatim vectorisation criterion shared by logic and style
• Concept-grain rule stated explicitly (one item per reader-facing API surface element, symmetric old/new — currently exists only by example)

Design input needed from @xuanguang-li

• Readability instrument v2 — the one critique that fully survived: docstring coverage inverts the framework's own exemplars (odu.py, the LOW example, measures 0.86; flagship-aiyagari style measures ~0.41–0.55 and caps at 2). Proposal: equation-traceability (per numbered equation, cite the single implementing expression; fraction traceable old vs new), with concept lists moving into evidence.json as cited slots (extends the M1 proposal on Land the lecture evaluation system (benchmark plugin 0.3.0: rubric v2, skill wired) #5)
• Extraction/replay verification — a mechanical step diffing extracted code and replayed call sequences against the lecture's actual cells; longer-term, evaluate executing the lecture itself at both refs (nbclient per-cell timings, PROJECT: QuantEcon benchmarking programme (code performance & execution) meta#335 telemetry) as the as-used source

Explicitly defended — no change

The efficiency ratio form (goal-failure semantics; log-rescoring changes no committed verdict), the weighted total (the four-gate alternative was shown to be a lossy projection fitted on the rubric's own calibration set), min() aggregation's Goodhart-resistant shape, and per-consequence multi-counting as a principle.

Relates to #4 (skill wiring — the v2 rubric changes what the skill drives) and #5 (the landing PR carrying the corrections).

🤖 Generated with Claude Code

[mmcky]

From the PR #5 review (the enforcement items were applied on the branch — see the response comment), five methodology items were deliberately deferred to this issue's design track rather than decided in a hardening pass:

Item	Decision to make
C2 — no-conversion floor asymmetry	The gate fires only when the baseline is sub-floor and the candidate is slower (rubric.py, no_conv). Triage decides from the baseline alone, so review and triage still disagree on the faster-but-immaterial case (e.g. 1.2× on a 0.035 s baseline buys 7 ms and gets a normal band). If the floor's principle is materiality, it should key on the baseline alone with the speedup only shaping the wording.
C5 — pin the as_used_runs pairing definition	run_all.py zips numpy run i with jax run i across independent fresh processes: the pairing is arbitrary, zip truncates silently if one side broke early, and median-of-ratios ≠ ratio-of-medians. Decide before any real runs data is committed — the first backfilled evidence bakes the definition into the regression baseline that CI now enforces.
C4 — fill_evidence.py	The pipeline measures into gitignored results/*.json then scores a hand-filled evidence.json from a previous run, with a field-name gap (as_used_speedup_runs vs as_used_runs) forcing manual transcription. validate_evidence() now catches omission but not a typo'd value; mechanical filling of the quantitative slots removes the error class.
D — comparative vs absolute	The verdict vocabulary is comparative (net regression / clear improvement) but most scored inputs are absolute properties of the candidate. Sharpest case: docstring_cov_new discards the delta static_metrics.json already records — 0.90→0.55 scores identically to 0.10→0.55. Option 1 (state in framework §1 which dimensions are comparative and why) costs a paragraph; option 2 (make it a delta) moves ge_arrow's readability score, so it is a v3 change.
C1's second half — split the sensitivity stamp	The perturbation walk flips measured observations (builds) and adjudicated judgements (checklist criteria) alike, so fragile conflates "different observation" with "different opinion". The floor half landed (robust-at-floor); the measured/adjudicated partition is recorded in framework §1 as a known limit for v3.

[mmcky]

Status after #5 merged — the engine half of this is done and shipped.

All four engine changes are in benchmark/scripts/scoring/ on main: the safety couplings derived inside score_all, the no-conversion verdict, the sensitivity stamp, and K-repeat as-used measurement. Both committed evidence files re-score to their documented anchors, and CI regenerates the scorecards from evidence and fails on any diff — so the couplings are now enforced by the build rather than by review attention. The synthetic fixtures under benchmark/references/fixtures/ pin the v2 paths, including the honest-evidence 4.2 case this issue was written around.

Also worth recording against the corrections of record above: the unconditional matches_under_x64 cap now applies on its own rather than only in combination, which was the remaining hole found in the PR #5 review.

Still open, unchanged:

• The documentation honesty pass (thresholds as policy choices with derivation notes, the fan-out paragraph, the explicit concept-grain rule). Some threshold-labelling work landed with Land the lecture evaluation system (benchmark plugin 0.3.0: rubric v2, skill wired) #5, but not enough of it that I would tick these boxes on your behalf — worth a read-through before checking them off.
• Both items needing @xuanguang-li's design input — the readability instrument v2 and the extraction/replay verification step. These are the real remainder of this issue.

One stale link: the standing reference to reviews/benchmark-design-2026-07-21-merged.md points at the docs-skills-usage branch, now deleted. It is on main: https://github.com/QuantEcon/skills/blob/main/reviews/benchmark-design-2026-07-21-merged.md — and rendered on the docs site is only the guide set, not reviews/, which stays in-repo as working records.

[mmcky]

Nothing in the rubric or the engine has moved since #5. benchmark/references/EVALUATION_FRAMEWORK.md and every file under benchmark/scripts/ are still at commit 8a854c6, none of the three open PRs (#30, #31, #32) touches scripts/ or references/, and every unchecked box above is still unchecked.

The materiality floor now has a measured counter-case committed to the repo. reviews/triage-run-wald_friedman-2026-07-27.md records a baseline as-used total of 5.586 s — median of three fresh processes (5.755 / 5.586 / 5.276), 5.6× over NO_CONVERSION_BASELINE_S — of which only ~4% is addressable by the conversion contemplated, an Amdahl ceiling of 1.045×. The C2 row of the deferred table above asks whether the gate should key on the baseline alone rather than on baseline-and-slower-candidate; this case asks a second question about the same constant — whether the quantity should be addressable time rather than the total. Both are NO_CONVERSION_BASELINE_S at rubric.py:113, and they are worth deciding together rather than one on each issue. The second is finding 4 on #14, unanswered since 2026-07-27.

The same run bears on "Extraction/replay verification" above. Step 1 of benchmark/skills/review-acceleration/SKILL.md currently asks in prose to "diff the extracted code and the replayed call sequence against the lecture's cells and fix mismatches". The run applied that as a hard gate — every extracted line and every replayed call must appear verbatim in the lecture source, compared right-stripped — and it caught construction drift on its first use. That is field evidence for the shape the mechanical step should take; the step itself is still unbuilt, so the box stays unchecked.

Whoever lands the documentation-honesty pass should plan on a version bump plus a matching benchmark/CHANGELOG.md entry in the same PR (#28). #30 is open and already claims 0.3.2.