From 714978d1d05655e8003b7060e3bcdc1f6b7c304b Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Fri, 7 Aug 2026 10:23:21 +1000 Subject: [PATCH 1/3] =?UTF-8?q?benchmark:=20triage=20first=20=E2=80=94=20a?= =?UTF-8?q?dvise=20before=20code=20is=20written;=20every=20output=20leads?= =?UTF-8?q?=20with=20the=20decision=20(0.4.0)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Triage — "should this lecture be converted at all?" — becomes the skill's front door: SKILL.md leads with it behind a Which-mode router, the frontmatter description opens with the advise use case so natural-language invocation matches the common question, and the README puts it first throughout. Review mode is unchanged and stays for the day a conversion PR exists again. Recommendation-first output everywhere (skills#14, finding 6): score.py prints VERDICT above the weighted total, labels the total "for the record", and the deciding-flip lines name the verdict they flip to before the recomputed number; the report format's TL;DR opens with the full verdict and the dimension table gains a verdict row. The tutorial's quoted output follows. No rubric, weight, gate, or scorecard-JSON change: the regression anchors and fixtures reproduce unchanged. Co-Authored-By: Claude Fable 5 --- .claude-plugin/marketplace.json | 2 +- benchmark/.claude-plugin/plugin.json | 2 +- benchmark/CHANGELOG.md | 12 +++++ benchmark/README.md | 36 +++++++-------- benchmark/scripts/scoring/score.py | 10 ++++- benchmark/skills/review-acceleration/SKILL.md | 45 +++++++++++-------- docs/tutorial-run-an-evaluation.md | 10 +++-- 7 files changed, 73 insertions(+), 44 deletions(-) diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 2ad0473..8448857 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -15,7 +15,7 @@ { "name": "benchmark", "source": "./benchmark", - "version": "0.3.2", + "version": "0.4.0", "description": "Benchmarking and acceleration-evaluation tools for QuantEcon lecture code" }, { diff --git a/benchmark/.claude-plugin/plugin.json b/benchmark/.claude-plugin/plugin.json index 85712d5..142159f 100644 --- a/benchmark/.claude-plugin/plugin.json +++ b/benchmark/.claude-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "benchmark", "description": "Benchmarking and acceleration-evaluation tools for QuantEcon lecture code", - "version": "0.3.2", + "version": "0.4.0", "author": { "name": "QuantEcon" } } diff --git a/benchmark/CHANGELOG.md b/benchmark/CHANGELOG.md index 77e785a..50892f2 100644 --- a/benchmark/CHANGELOG.md +++ b/benchmark/CHANGELOG.md @@ -6,6 +6,18 @@ Versions are [semver](https://semver.org) as a user of this plugin experiences i Repository: [QuantEcon/skills](https://github.com/QuantEcon/skills) ([every commit that touched this plugin](https://github.com/QuantEcon/skills/commits/main/benchmark)). How a release is made: [developing-skills § Versioning and releases](https://github.com/QuantEcon/skills/blob/main/docs/developing-skills.md#versioning-and-releases). +## 0.4.0 — 2026-08-07 + +Triage becomes the front door, and every output leads with the decision. The reframing follows the maintainers' direction — the product most wanted is "look at a lecture and advise whether a JAX upgrade is recommended" — and the measured record agrees: in every evaluation to date (ge_arrow, markov_asset, wald_friedman, and the 2026-08-06 ge_arrow re-run on [skills#10](https://github.com/QuantEcon/skills/issues/10)) the recommendation was decided by the triage-layer instruments — the as-used baseline and what a conversion could reach — and never moved by the scorecard on top. Review mode is unchanged and stays: it is the mode that caught markov_asset's masked build defect, and it applies the day a conversion PR exists. + +**Changed** + +- `SKILL.md` leads with triage — the no-candidate "should this lecture be converted?" question — behind a "Which mode" router, with review as the second mode. The frontmatter description now opens with the advise use case, so natural-language invocation matches the common question. Review-mode content is unchanged. +- The scorer's printed output and the report format lead with the verdict. `score.py` prints `VERDICT:` above the weighted total, labels the total "for the record", and the deciding-flip lines name the verdict they flip to before the recomputed number (previously `⇒ total 2.30, …`, which two careful readers in a row took as the headline — [skills#14, finding 6](https://github.com/QuantEcon/skills/issues/14)). The report's TL;DR opens with the full verdict and carries the score alongside as candidate quality for the record; the dimension table gains a verdict row so it still carries the decision when quoted on its own. +- `README.md` puts triage first throughout — the mode table, the invocation examples, and the mode sections — and states that triage builds no candidate: it measures the lecture as it stands and bounds what a conversion could deliver. + +Nothing in the rubric, weights, gates, or scorecard JSON changed: the regression anchors (2.85 / 2.25) and the fixtures reproduce unchanged. + ## 0.3.2 — 2026-08-03 **Fixed** diff --git a/benchmark/README.md b/benchmark/README.md index 3ecd364..d71292c 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -2,39 +2,25 @@ Evaluation tooling for QuantEcon lecture code rewrites — the question it answers is never "is JAX faster?" but **"does this implementation earn its place in this lecture?"** Lectures are teaching materials first and programs second; the plugin's rubric weights readability (0.25) above efficiency (0.15) on purpose. -One skill, two modes: +One skill, two modes — triage is the front door, since most questions arrive before any code is written: | Mode | Question | Needs | Produces | |---|---|---|---| +| **Triage** | Is this lecture worth converting at all? | the existing lecture only | A recommendation with the binding constraint named | | **Review** | Did this conversion PR improve the lecture? | baseline + candidate implementations | A scored report with a merge recommendation | -| **Triage** | Is this lecture worth converting at all? | the existing lecture only | A predicted verdict band with the binding constraint named | Status: operational for workspace runs since v0.3.0, which wired the skill to the scoring engine — what shipped in which release is in [CHANGELOG.md](CHANGELOG.md), and remaining plan items are in [skills#4](https://github.com/QuantEcon/skills/issues/4). The system was developed and validated by [@xuanguang-li](https://github.com/xuanguang-li) on [lecture-python.myst#717](https://github.com/QuantEcon/lecture-python.myst/pull/717) and [#654](https://github.com/QuantEcon/lecture-python.myst/pull/654). ## Using the skill ``` -/benchmark:review-acceleration # review mode /benchmark:review-acceleration should we convert ? # triage mode +/benchmark:review-acceleration # review mode ``` -### Review mode — what you get - -The skill follows the measure → record-evidence → score contract ([scripts/README.md](scripts/README.md)): it extracts both implementations verbatim from the lecture's code cells, adapts the measurement templates, runs them, fills `evidence.json` with cited answers, and lets the engine compute the verdict — **no score is ever typed by hand**. The session shows the engine's derivation table (every score with the measured number and threshold band that produced it), and the final report follows the worked examples' format: - -1. **TL;DR** — weighted score, verdict band, the decisive facts in one paragraph -2. **Dimension table** — weight / score / weighted contribution / one-line driver each -3. **What changed** — before/after implementation shape -4. **Evidence by dimension** — `max|Δ|` in both dtype regimes, prerequisite-concept and docstring deltas, the as-used vs warm timing table, crossover-n, recompile audit -5. **Recommendation** — a must-fix list where each item is tagged with the dimension it lifts, plus where the score lands after fixes - -See [references/examples/ge_arrow/ge_arrow_REPORT.md](references/examples/ge_arrow/ge_arrow_REPORT.md) (2.85/5, no-conversion; candidate band mixed/wash) and [references/examples/markov_asset/markov_asset_REPORT.md](references/examples/markov_asset/markov_asset_REPORT.md) (2.25/5, no-conversion + gated net regression) for complete real reports. Verdict bands, the v2 verdict gates / no-conversion rule / sensitivity stamp, weights, and scoring anchors: [references/EVALUATION_FRAMEWORK.md](references/EVALUATION_FRAMEWORK.md) §1–2. - -**The one rule to remember:** warm-only speedups are never the headline. The ge_arrow case measured 1.4–4.8× faster warm and **45× slower as-used** — the as-used number (fresh process, actual problem sizes, compile time included) decides the efficiency score. - ### Triage mode — before any code is written -Four checks, using only the existing lecture: +Four checks, using only the existing lecture — no candidate is built: 1. **Baseline as-used total** — replay the lecture's real call sequence (the NumPy half of an `as_used_total.py` template). This bounds the entire possible win: a lecture whose compute totals 30ms has nothing to give. 2. **Workload-pattern match** — against the two calibrated poles: **aiyagari-shaped** (large fixed-shape arrays, many re-solves, stable static args → measured ~24× as-used win) vs **ge_arrow-shaped** (tiny models, fresh static args per call → measured ~45× as-used loss). @@ -55,6 +41,20 @@ The baseline totals above are the **triage-time** measurements taken on 2026-07- Scope limit, confirmed by the same test: triage predicts whether the prize is worth pursuing — it cannot predict conversion-quality outcomes (markov_asset's masked `err.throw()` defect was a property of the PR, invisible to triage). Note also that this validation is **in-sample** — the three cases are the ones the thresholds were calibrated on; out-of-sample validation accumulates as fresh lectures are triaged. +### Review mode — what you get + +The skill follows the measure → record-evidence → score contract ([scripts/README.md](scripts/README.md)): it extracts both implementations verbatim from the lecture's code cells, adapts the measurement templates, runs them, fills `evidence.json` with cited answers, and lets the engine compute the verdict — **no score is ever typed by hand**. The session shows the engine's derivation table (every score with the measured number and threshold band that produced it), and the final report follows the worked examples' format, leading with the decision: + +1. **TL;DR** — the *full* verdict first (gate / no-conversion / sensitivity included), the decisive facts, and the weighted score alongside as candidate quality for the record +2. **Dimension table** — a verdict row, then weight / score / weighted contribution / one-line driver each, so the table still carries the decision when quoted on its own +3. **What changed** — before/after implementation shape +4. **Evidence by dimension** — `max|Δ|` in both dtype regimes, prerequisite-concept and docstring deltas, the as-used vs warm timing table, crossover-n, recompile audit +5. **Recommendation** — a must-fix list where each item is tagged with the dimension it lifts, plus where the score lands after fixes + +See [references/examples/ge_arrow/ge_arrow_REPORT.md](references/examples/ge_arrow/ge_arrow_REPORT.md) (2.85/5, no-conversion; candidate band mixed/wash) and [references/examples/markov_asset/markov_asset_REPORT.md](references/examples/markov_asset/markov_asset_REPORT.md) (2.25/5, no-conversion + gated net regression) for complete real reports. Verdict bands, the v2 verdict gates / no-conversion rule / sensitivity stamp, weights, and scoring anchors: [references/EVALUATION_FRAMEWORK.md](references/EVALUATION_FRAMEWORK.md) §1–2. + +**The one rule to remember:** warm-only speedups are never the headline. The ge_arrow case measured 1.4–4.8× faster warm and **45× slower as-used** — the as-used number (fresh process, actual problem sizes, compile time included) decides the efficiency score. + ## Manual usage (no skill) The full recipe is in [scripts/README.md](scripts/README.md) ("Evaluating a new lecture"); quickstart from this directory: diff --git a/benchmark/scripts/scoring/score.py b/benchmark/scripts/scoring/score.py index 8dbf01e..142f8a3 100644 --- a/benchmark/scripts/scoring/score.py +++ b/benchmark/scripts/scoring/score.py @@ -127,8 +127,13 @@ def main(lecture_dir): f"{r['score']:>3d} {r['weighted']:>5.2f}") print(f" └ {r['reason']}") print("-" * 78) - print(f"{'WEIGHTED TOTAL':34s} {'':11s} {'':>4s} {'':>3s} {result['total']:>5.2f}") + # The verdict prints before the number: the two are different layers (the + # total measures polish, the verdict carries the decision), and a reader + # who takes away one line must take away the recommendation. Both the + # system's designer and its sponsor read the total as the headline when it + # came first — skills#14, finding 6. print(f"VERDICT: {result['verdict']}") + print(f"{'WEIGHTED TOTAL (for the record)':34s} {'':11s} {'':>4s} {'':>3s} {result['total']:>5.2f}") skipped = sens["perturbations_skipped"] print(f"SENSITIVITY: {sens['stamp']} " f"({sens['perturbations_tested']} single-input perturbations scored" @@ -138,7 +143,8 @@ def main(lecture_dir): print(f" └ {sens['stamp_note']}") for fl in sens["deciding_flips"]: print(f" └ {fl['input']}: {fl['from']} → {fl['to']} " - f"⇒ total {fl['total']:.2f}, {fl['outcome']}") + f"⇒ verdict {fl['outcome']} " + f"(recomputed total, for the record: {fl['total']:.2f})") out = { "lecture": lecture, diff --git a/benchmark/skills/review-acceleration/SKILL.md b/benchmark/skills/review-acceleration/SKILL.md index 90049f4..fcceab1 100644 --- a/benchmark/skills/review-acceleration/SKILL.md +++ b/benchmark/skills/review-acceleration/SKILL.md @@ -1,11 +1,24 @@ --- name: review-acceleration -description: Review whether an accelerated implementation (JAX, Numba) of QuantEcon lecture code actually improves the lecture. Measures as-used performance (fresh process, JIT compile time included), numerical fidelity (float32/float64), and readability, then produces a weighted scored report with a merge recommendation. Use when reviewing a NumPy→JAX/Numba conversion PR or deciding whether to convert a lecture. +description: Advise whether a QuantEcon lecture is worth converting to an accelerated implementation (JAX, Numba), or review whether an existing conversion PR actually improves the lecture. Triage mode needs only the lecture — it measures the as-used baseline, bounds the possible win, and recommends convert or don't-convert with the binding constraint named. Review mode measures a candidate against the baseline (as-used performance in a fresh process with JIT compile time included, numerical fidelity under float32/float64, readability) and produces a scored report with a merge recommendation. Use when deciding whether to convert a lecture to JAX, or when reviewing a NumPy→JAX/Numba conversion PR. --- # review-acceleration -> **Status: evaluation system landed (rubric v2); skill wired for workspace runs.** The system was developed and validated by @xuanguang-li on [QuantEcon/lecture-python.myst#717](https://github.com/QuantEcon/lecture-python.myst/pull/717) and [#654](https://github.com/QuantEcon/lecture-python.myst/pull/654) and now lives in this plugin: the rubric in [`references/EVALUATION_FRAMEWORK.md`](../../references/EVALUATION_FRAMEWORK.md), the deterministic scoring engine in `scripts/scoring/`, and two complete worked evaluations in `references/examples/`. Rubric v2 (verdict gates, no-conversion, sensitivity stamp, K-repeat as-used) implements the surviving critiques of the 2026-07-21 design review. Tracking: [QuantEcon/skills#4](https://github.com/QuantEcon/skills/issues/4), [QuantEcon/meta#335](https://github.com/QuantEcon/meta/issues/335) (workstream B). +> **Status: evaluation system landed (rubric v2); skill wired for workspace runs; triage-first since 0.4.0.** The system was developed and validated by @xuanguang-li on [QuantEcon/lecture-python.myst#717](https://github.com/QuantEcon/lecture-python.myst/pull/717) and [#654](https://github.com/QuantEcon/lecture-python.myst/pull/654) and now lives in this plugin: the rubric in [`references/EVALUATION_FRAMEWORK.md`](../../references/EVALUATION_FRAMEWORK.md), the deterministic scoring engine in `scripts/scoring/`, and two complete worked evaluations in `references/examples/`. Rubric v2 (verdict gates, no-conversion, sensitivity stamp, K-repeat as-used) implements the surviving critiques of the 2026-07-21 design review. Tracking: [QuantEcon/skills#4](https://github.com/QuantEcon/skills/issues/4), [QuantEcon/meta#335](https://github.com/QuantEcon/meta/issues/335) (workstream B). + +## Which mode + +Two questions, two modes — pick by whether a candidate implementation exists: + +- **No candidate — "should this lecture be converted at all?"** → [Triage](#triage--should-this-lecture-be-converted-start-here). The common case: it needs only the existing lecture, builds nothing, and answers with a recommendation and the constraint that binds it. +- **A conversion PR or candidate branch exists — "did it improve the lecture?"** → [Review](#review--did-this-conversion-improve-the-lecture). The full measured evaluation with a merge recommendation. + +Whichever mode runs, **the recommendation is the headline of everything this skill produces** — a reader who takes away one line must take away the decision, with any score carried alongside as candidate quality for the record, never the other way around. + +## Guiding principle + +QuantEcon lectures are teaching materials first and programs second. A rewrite that is faster or more modern but harder for a learner to read, or that silently changes published numbers, is not an improvement. "Uses JAX" is never a goal in itself — the accelerated implementation must earn its place on each lecture. ## Where things live at run time @@ -21,13 +34,20 @@ When this skill runs from the installed plugin, the plugin's files are **read-on Never write into the plugin directory. Scoring works on any directory: `python ${CLAUDE_PLUGIN_ROOT}/scripts/scoring/score.py benchmark-eval/`. The scaffolded `run_all.py` reads `CLAUDE_PLUGIN_ROOT` from the environment to find the shared engine — export it (or keep the adapted script's path pointing at the plugin) before running the pipeline. -**Preconditions to verify before starting** (fail loudly, don't improvise silently): a checkout of the lecture repo with both refs fetchable (baseline, usually `main`, and the candidate branch); a Python environment with `jax`, `numpy`, and the lecture's imports (the reference runs used the `quantecon` conda env); CPU-only is the calibrated regime. Record the environment via the provenance stamp — `run_all.py` does this automatically, including failed-step titles, so a partial run cannot claim full provenance. +**Preconditions to verify before starting** (fail loudly, don't improvise silently): a checkout of the lecture repo — for review mode, with both refs fetchable (baseline, usually `main`, and the candidate branch); a Python environment with `jax`, `numpy`, and the lecture's imports (the reference runs used the `quantecon` conda env); CPU-only is the calibrated regime. Record the environment via the provenance stamp — `run_all.py` does this automatically, including failed-step titles, so a partial run cannot claim full provenance. -## Guiding principle +## Triage — should this lecture be converted? (start here) -QuantEcon lectures are teaching materials first and programs second. A rewrite that is faster or more modern but harder for a learner to read, or that silently changes published numbers, is not an improvement. "Uses JAX" is never a goal in itself — the accelerated implementation must earn its place on each lecture. +When the question is "should this lecture be converted at all," run the prospective subset — only the existing lecture is needed, and **no candidate is built**: the skill measures the lecture as it stands and bounds what a conversion could deliver, which is what makes the advice cheap enough to ask for routinely. + +1. **Baseline as-used total**: adapt just the baseline half of an `as_used_total.py` template and replay the lecture's real call sequence — this bounds the maximum possible win (a 30 ms lecture has nothing to give). +2. **Pattern-match** against the calibrated poles: aiyagari-shaped (large fixed shapes, many re-solves, stable static args → ~24× win) vs ge_arrow-shaped (tiny models, fresh static args per call → ~45× loss). +3. **Crossover check**: the lecture's problem sizes vs warm crossover-n. +4. **Readability-cost forecast**: which prerequisite concepts the conversion would force. + +Decision rule from the weights: efficiency (0.15) gains at most +0.30 weighted; readability (0.25) losing two bands costs −0.50 — a conversion that costs meaningful readability cannot break even on speed alone, and structural wins are usually achievable in the baseline library. Report a predicted verdict band with the binding constraint named, not a scorecard. Validated 2026-07-21: blind triage on ge_arrow, markov_asset (both sub-second baselines → don't convert) and the aiyagari pattern (~54 s → convert) reproduced all three known verdicts, from the triage-time baseline measurements recorded in the plugin README; triage cannot predict conversion-quality defects (markov_asset's build bug), and must say so. Rubric v2 closes the loop from the review side: when a full evaluation's efficiency evidence shows the don't-convert profile (baseline under the 1 s floor, candidate slower as-used), the scorecard itself emits the **no-conversion** verdict — review and triage can no longer disagree on that question. -## Procedure +## Review — did this conversion improve the lecture? Given a baseline implementation (usually `main`) and a candidate (usually a PR branch) for one lecture, follow the measure → record-evidence → score contract in [`scripts/README.md`](../../scripts/README.md) — **scores are never typed by hand**: @@ -35,21 +55,10 @@ Given a baseline implementation (usually `main`) and a candidate (usually a PR b 2. **Measure** — `run_all.py`: equivalence under the default dtype AND `jax_enable_x64` (report `max|Δ|` per regime); static metrics (prerequisite concepts, docstring coverage); the **as-used benchmark** — replay the lecture's *actual* solver call sequence at its *actual* sizes in a fresh interpreter so trace/compile time counts, repeated ≥3 times per side with the **median** as the headline (`as_used_speedup = baseline median / candidate median`), with warm timings alongside (never alone), a crossover-n scaling curve, and a recompile audit. A provenance stamp (`results/env.json`, generated per-run) records the environment and any failed steps. 3. **Record evidence** — fill `evidence.json` from the results: measured numbers into the quantitative slots with sources (including `baseline_as_used_seconds` and the per-run `as_used_runs`); each structural checklist item answered true/false **with a citation to the diff**. This file is the only place judgement is recorded. 4. **Score** — `python ${CLAUDE_PLUGIN_ROOT}/scripts/scoring/score.py benchmark-eval/` computes all seven dimensions and the weighted total deterministically. The weights, threshold anchors, and verdict bands are defined in [`references/EVALUATION_FRAMEWORK.md`](../../references/EVALUATION_FRAMEWORK.md) §1–2 and machine-encoded in `scripts/scoring/rubric.py` — never restate or re-derive them here. v2 outputs you must carry into the report verbatim: the **verdict gate** (correctness 1/2 caps the band), the **no-conversion** verdict (don't-convert profile beats polish), and the **sensitivity stamp** (robust / robust-at-floor / fragile, with the deciding flips). Carry the stamp as printed — *robust-at-floor* means the outcome held only because it is already in the bottom band and could not get worse, so never report it as *robust*. Because one stamp currently covers both measurement and judgement perturbations ([framework §1](../../references/EVALUATION_FRAMEWORK.md)), quote the deciding-flip list rather than resting the report's confidence on the word alone. -5. **Report** — write `_REPORT.md` from the scorecard + evidence, following the worked examples' format: TL;DR with the weighted score and the *full* verdict (including gate/no-conversion/sensitivity), the dimension table with drivers, evidence per dimension, and a must-fix list mapping each recommendation to the dimension it lifts. +5. **Report** — write `_REPORT.md` from the scorecard + evidence, following the worked examples' format and **leading with the decision**: the TL;DR opens with the *full* verdict (including gate/no-conversion/sensitivity) and carries the weighted score alongside as candidate quality for the record. Then the dimension table with drivers — including a verdict row, so the table still carries the decision when it is quoted on its own — evidence per dimension, and a must-fix list mapping each recommendation to the dimension it lifts. The scorecard's two outputs are different layers: the total measures the candidate's polish, the verdict carries the recommendation, and a report must never present the number where the decision belongs (skills#14, finding 6: careful readers took the total for the headline twice). Never present warm-only speedups as the headline — the ge_arrow case measured 1.4–4.8× faster warm and ~45× slower as-used. -## Triage mode (no candidate yet) - -When the question is "should this lecture be converted at all," run the prospective subset — only the existing lecture is needed: - -1. **Baseline as-used total**: adapt just the baseline half of an `as_used_total.py` template and replay the lecture's real call sequence — this bounds the maximum possible win (a 30 ms lecture has nothing to give). -2. **Pattern-match** against the calibrated poles: aiyagari-shaped (large fixed shapes, many re-solves, stable static args → ~24× win) vs ge_arrow-shaped (tiny models, fresh static args per call → ~45× loss). -3. **Crossover check**: the lecture's problem sizes vs warm crossover-n. -4. **Readability-cost forecast**: which prerequisite concepts the conversion would force. - -Decision rule from the weights: efficiency (0.15) gains at most +0.30 weighted; readability (0.25) losing two bands costs −0.50 — a conversion that costs meaningful readability cannot break even on speed alone, and structural wins are usually achievable in the baseline library. Report a predicted verdict band with the binding constraint named, not a scorecard. Validated 2026-07-21: blind triage on ge_arrow, markov_asset (both sub-second baselines → don't convert) and the aiyagari pattern (~54 s → convert) reproduced all three known verdicts, from the triage-time baseline measurements recorded in the plugin README; triage cannot predict conversion-quality defects (markov_asset's build bug), and must say so. Rubric v2 closes the loop from the review side: when a full evaluation's efficiency evidence shows the don't-convert profile (baseline under the 1 s floor, candidate slower as-used), the scorecard itself emits the **no-conversion** verdict — review and triage can no longer disagree on that question. - ## Calibration baseline (regression anchors) The two worked evaluations in `references/examples/` are the validation baseline — re-running their pipelines must reproduce these verdicts. Confirmed end-to-end 2026-07-22: a fresh-checkout workspace run of ge_arrow (#717, base `8cfba4c`) on a different machine and jax **0.10.1** (reference: 0.4.35) reproduced 2.85 / no-conversion / fragile with the same deciding flips — every measured quantity moved only within its band. Evidence files record `source_pr` + base/head SHAs: diff --git a/docs/tutorial-run-an-evaluation.md b/docs/tutorial-run-an-evaluation.md index e3e93d7..75a9fd9 100644 --- a/docs/tutorial-run-an-evaluation.md +++ b/docs/tutorial-run-an-evaluation.md @@ -80,17 +80,19 @@ python $CLAUDE_PLUGIN_ROOT/scripts/scoring/score.py benchmark-eval/ge_arrow No score is ever typed by hand — the engine computes all seven dimensions and prints the derivation of each. The validation run's tail: ``` -WEIGHTED TOTAL 2.85 VERDICT: no-conversion — the baseline as-used total 0.0292 s is under the 1 s materiality floor and the candidate is slower as-used (0.0251×): this lecture should not be converted, whatever the candidate's polish. Candidate quality for the record: 2.85/5, mixed/wash +WEIGHTED TOTAL (for the record) 2.85 SENSITIVITY: fragile (29 single-input perturbations scored) - └ quantitative.correctness.builds: True → False ⇒ total 2.30, ... - └ quantitative.correctness.matches_under_x64: True → False ⇒ total 2.30, ... - └ structural.logic_design.criteria.good_algorithmic_choices: False → True ⇒ total 3.00, ... + └ quantitative.correctness.builds: True → False ⇒ verdict no-conversion; net regression — ... (recomputed total, for the record: 2.30) + └ quantitative.correctness.matches_under_x64: True → False ⇒ verdict no-conversion; net regression — ... (recomputed total, for the record: 2.30) + └ structural.logic_design.criteria.good_algorithmic_choices: False → True ⇒ verdict no-conversion; net positive — ... (recomputed total, for the record: 3.00) ``` +The verdict prints first, and the deciding flips name the verdict they flip to before the recomputed number: the total and the verdict are different layers — the total measures the candidate's polish, the verdict carries the decision — and the output order makes sure a reader who takes away one line takes away the recommendation. + Three things to read off a v2 scorecard beyond the total: the **verdict gate** (broken correctness caps the band regardless of polish), the **no-conversion** verdict (a lecture with nothing to gain shouldn't be converted, however good the candidate), and the **sensitivity stamp** (would any single contestable input flip the outcome? here: yes, three would — the scorecard says so instead of hiding it). ## Step 5 — cross-compare From 9ef93f46ec0de4780212ec80f2695e26d9b01831 Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Fri, 7 Aug 2026 10:28:19 +1000 Subject: [PATCH 2/3] benchmark: triage cites the manual's JAX page as its decision criteria MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The canonical "when to use JAX / when not to" guidance already exists — manual.quantecon.org/styleguide/jax.html, including Converting from Numba § Decide first. Triage now cites it rather than restating it, and frames its four checks as the measurement layer that tests whether the page's criteria hold for a given lecture. Co-Authored-By: Claude Fable 5 --- benchmark/CHANGELOG.md | 1 + benchmark/README.md | 2 +- benchmark/skills/review-acceleration/SKILL.md | 2 ++ 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/benchmark/CHANGELOG.md b/benchmark/CHANGELOG.md index 50892f2..8b5cc39 100644 --- a/benchmark/CHANGELOG.md +++ b/benchmark/CHANGELOG.md @@ -15,6 +15,7 @@ Triage becomes the front door, and every output leads with the decision. The ref - `SKILL.md` leads with triage — the no-candidate "should this lecture be converted?" question — behind a "Which mode" router, with review as the second mode. The frontmatter description now opens with the advise use case, so natural-language invocation matches the common question. Review-mode content is unchanged. - The scorer's printed output and the report format lead with the verdict. `score.py` prints `VERDICT:` above the weighted total, labels the total "for the record", and the deciding-flip lines name the verdict they flip to before the recomputed number (previously `⇒ total 2.30, …`, which two careful readers in a row took as the headline — [skills#14, finding 6](https://github.com/QuantEcon/skills/issues/14)). The report's TL;DR opens with the full verdict and carries the score alongside as candidate quality for the record; the dimension table gains a verdict row so it still carries the decision when quoted on its own. - `README.md` puts triage first throughout — the mode table, the invocation examples, and the mode sections — and states that triage builds no candidate: it measures the lecture as it stands and bounds what a conversion could deliver. +- Triage now names its canonical decision criteria: the manual's JAX style page ([when to use JAX, when not to](https://manual.quantecon.org/styleguide/jax.html), including *Converting from Numba § Decide first*), cited rather than restated. The skill's four checks are framed as the measurement layer that tests whether those criteria hold for a given lecture — "a real bottleneck" is a claim the as-used baseline and pattern match establish or refute, while the page's "teaching JAX itself" criterion is editorial and stays a maintainer call. Nothing in the rubric, weights, gates, or scorecard JSON changed: the regression anchors (2.85 / 2.25) and the fixtures reproduce unchanged. diff --git a/benchmark/README.md b/benchmark/README.md index d71292c..af17291 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -20,7 +20,7 @@ Status: operational for workspace runs since v0.3.0, which wired the skill to th ### Triage mode — before any code is written -Four checks, using only the existing lecture — no candidate is built: +The decision criteria are canonical in the manual's JAX style page — [when to use JAX, when not to](https://manual.quantecon.org/styleguide/jax.html) — and the advice cites them; the skill's contribution is the measurement layer that tests whether they hold. Four checks, using only the existing lecture — no candidate is built: 1. **Baseline as-used total** — replay the lecture's real call sequence (the NumPy half of an `as_used_total.py` template). This bounds the entire possible win: a lecture whose compute totals 30ms has nothing to give. 2. **Workload-pattern match** — against the two calibrated poles: **aiyagari-shaped** (large fixed-shape arrays, many re-solves, stable static args → measured ~24× as-used win) vs **ge_arrow-shaped** (tiny models, fresh static args per call → measured ~45× as-used loss). diff --git a/benchmark/skills/review-acceleration/SKILL.md b/benchmark/skills/review-acceleration/SKILL.md index fcceab1..d521bdd 100644 --- a/benchmark/skills/review-acceleration/SKILL.md +++ b/benchmark/skills/review-acceleration/SKILL.md @@ -40,6 +40,8 @@ Never write into the plugin directory. Scoring works on any directory: `python $ When the question is "should this lecture be converted at all," run the prospective subset — only the existing lecture is needed, and **no candidate is built**: the skill measures the lecture as it stands and bounds what a conversion could deliver, which is what makes the advice cheap enough to ask for routinely. +The decision criteria themselves are canonical in the manual's JAX style page — [when to use JAX, when not to](https://manual.quantecon.org/styleguide/jax.html), including *Converting from Numba § Decide first* — cite them in the advice, never restate them. The checks below are the measurement layer that tests whether the page's criteria hold for this lecture: whether there is "a real bottleneck" is exactly what checks 1–2 establish or refute, and the sequential-vs-vectorizable question is check 2's pattern match. The page's "teaching JAX itself" criterion is editorial, not measurable — when it might apply, say so and leave it as the maintainer's call. + 1. **Baseline as-used total**: adapt just the baseline half of an `as_used_total.py` template and replay the lecture's real call sequence — this bounds the maximum possible win (a 30 ms lecture has nothing to give). 2. **Pattern-match** against the calibrated poles: aiyagari-shaped (large fixed shapes, many re-solves, stable static args → ~24× win) vs ge_arrow-shaped (tiny models, fresh static args per call → ~45× loss). 3. **Crossover check**: the lecture's problem sizes vs warm crossover-n. From 754df8c63601bbedb8bb17b0338707510a9cad2d Mon Sep 17 00:00:00 2001 From: Matt McKay Date: Fri, 7 Aug 2026 10:42:13 +1000 Subject: [PATCH 3/3] benchmark: state the triage decision rule qualitatively; standardize on convert / don't-convert MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The rule re-derived numbers from the rubric weights (0.15/0.25, +0.30, −0.50) in SKILL.md and README.md while the skill's own scoring step says weights live only in EVALUATION_FRAMEWORK.md — a drift risk if the rubric is recalibrated. Both places now state the conclusion and point at the framework. Triage's outcome vocabulary is convert / don't-convert throughout, replacing the review-mode "predicted verdict band". From Copilot's review of this PR. Co-Authored-By: Claude Fable 5 --- benchmark/CHANGELOG.md | 1 + benchmark/README.md | 2 +- benchmark/skills/review-acceleration/SKILL.md | 2 +- 3 files changed, 3 insertions(+), 2 deletions(-) diff --git a/benchmark/CHANGELOG.md b/benchmark/CHANGELOG.md index 8b5cc39..851fdc4 100644 --- a/benchmark/CHANGELOG.md +++ b/benchmark/CHANGELOG.md @@ -16,6 +16,7 @@ Triage becomes the front door, and every output leads with the decision. The ref - The scorer's printed output and the report format lead with the verdict. `score.py` prints `VERDICT:` above the weighted total, labels the total "for the record", and the deciding-flip lines name the verdict they flip to before the recomputed number (previously `⇒ total 2.30, …`, which two careful readers in a row took as the headline — [skills#14, finding 6](https://github.com/QuantEcon/skills/issues/14)). The report's TL;DR opens with the full verdict and carries the score alongside as candidate quality for the record; the dimension table gains a verdict row so it still carries the decision when quoted on its own. - `README.md` puts triage first throughout — the mode table, the invocation examples, and the mode sections — and states that triage builds no candidate: it measures the lecture as it stands and bounds what a conversion could deliver. - Triage now names its canonical decision criteria: the manual's JAX style page ([when to use JAX, when not to](https://manual.quantecon.org/styleguide/jax.html), including *Converting from Numba § Decide first*), cited rather than restated. The skill's four checks are framed as the measurement layer that tests whether those criteria hold for a given lecture — "a real bottleneck" is a claim the as-used baseline and pattern match establish or refute, while the page's "teaching JAX itself" criterion is editorial and stays a maintainer call. +- The triage decision rule no longer re-derives numbers from the rubric weights: it states the conclusion qualitatively and points to `references/EVALUATION_FRAMEWORK.md`, which the skill's own scoring step already said was the only place weights live. Triage's outcome vocabulary is standardized on **convert / don't-convert** in both `SKILL.md` and `README.md`. (Caught by Copilot's review of the 0.4.0 PR.) Nothing in the rubric, weights, gates, or scorecard JSON changed: the regression anchors (2.85 / 2.25) and the fixtures reproduce unchanged. diff --git a/benchmark/README.md b/benchmark/README.md index af17291..2683b5e 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -27,7 +27,7 @@ The decision criteria are canonical in the manual's JAX style page — [when to 3. **Crossover comparison** — the lecture's problem sizes vs the warm crossover-n from the scaling data. 4. **Readability-cost forecast** — which concepts the conversion would force on readers (static args, `lax` carries, checkify, the float32/x64 distinction), against the prerequisite-concept bands. -Then the decision rule that falls out of the rubric weights: efficiency (0.15) can gain at most +0.30 weighted (band 3→5), while readability (0.25) losing two bands costs −0.50 — **a conversion that costs meaningful readability cannot break even on speed alone**; it must also win on logic & design and ergonomics, and those structural wins are usually achievable in plain NumPy. +Then the decision rule ([EVALUATION_FRAMEWORK.md §1](references/EVALUATION_FRAMEWORK.md)): the rubric weights readability above efficiency on purpose, so **a conversion that costs meaningful readability cannot break even on speed alone** — the weight algebra behind that lives in the framework, not here; it must also win on logic & design and ergonomics, and those structural wins are usually achievable in plain NumPy. **Validation (2026-07-21):** triage applied blind (baseline-side data only) to the three known cases reproduces every known verdict: diff --git a/benchmark/skills/review-acceleration/SKILL.md b/benchmark/skills/review-acceleration/SKILL.md index d521bdd..e7a6482 100644 --- a/benchmark/skills/review-acceleration/SKILL.md +++ b/benchmark/skills/review-acceleration/SKILL.md @@ -47,7 +47,7 @@ The decision criteria themselves are canonical in the manual's JAX style page 3. **Crossover check**: the lecture's problem sizes vs warm crossover-n. 4. **Readability-cost forecast**: which prerequisite concepts the conversion would force. -Decision rule from the weights: efficiency (0.15) gains at most +0.30 weighted; readability (0.25) losing two bands costs −0.50 — a conversion that costs meaningful readability cannot break even on speed alone, and structural wins are usually achievable in the baseline library. Report a predicted verdict band with the binding constraint named, not a scorecard. Validated 2026-07-21: blind triage on ge_arrow, markov_asset (both sub-second baselines → don't convert) and the aiyagari pattern (~54 s → convert) reproduced all three known verdicts, from the triage-time baseline measurements recorded in the plugin README; triage cannot predict conversion-quality defects (markov_asset's build bug), and must say so. Rubric v2 closes the loop from the review side: when a full evaluation's efficiency evidence shows the don't-convert profile (baseline under the 1 s floor, candidate slower as-used), the scorecard itself emits the **no-conversion** verdict — review and triage can no longer disagree on that question. +Decision rule ([framework §1](../../references/EVALUATION_FRAMEWORK.md)): the rubric weights readability above efficiency on purpose, so a conversion that costs meaningful readability cannot break even on speed alone — the weight algebra behind that is the framework's, not restated here — and structural wins are usually achievable in the baseline library. Report a recommendation — **convert or don't-convert** — with the binding constraint named, not a scorecard. Validated 2026-07-21: blind triage on ge_arrow, markov_asset (both sub-second baselines → don't convert) and the aiyagari pattern (~54 s → convert) reproduced all three known verdicts, from the triage-time baseline measurements recorded in the plugin README; triage cannot predict conversion-quality defects (markov_asset's build bug), and must say so. Rubric v2 closes the loop from the review side: when a full evaluation's efficiency evidence shows the don't-convert profile (baseline under the 1 s floor, candidate slower as-used), the scorecard itself emits the **no-conversion** verdict — review and triage can no longer disagree on that question. ## Review — did this conversion improve the lecture?