benchmark: triage first — advise before code is written; every output leads with the decision (0.4.0)

.claude-plugin/marketplace.json

@@ -15,7 +15,7 @@
     {
       "name": "benchmark",
       "source": "./benchmark",
-      "version": "0.3.2",
+      "version": "0.4.0",
       "description": "Benchmarking and acceleration-evaluation tools for QuantEcon lecture code"
     },
     {

benchmark/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "benchmark",
   "description": "Benchmarking and acceleration-evaluation tools for QuantEcon lecture code",
-  "version": "0.3.2",
+  "version": "0.4.0",
   "author": { "name": "QuantEcon" }
 }

benchmark/CHANGELOG.md

@@ -6,6 +6,20 @@ Versions are [semver](https://semver.org) as a user of this plugin experiences i
 
 Repository: [QuantEcon/skills](https://github.com/QuantEcon/skills) ([every commit that touched this plugin](https://github.com/QuantEcon/skills/commits/main/benchmark)). How a release is made: [developing-skills § Versioning and releases](https://github.com/QuantEcon/skills/blob/main/docs/developing-skills.md#versioning-and-releases).
 
+## 0.4.0 — 2026-08-07
+
+Triage becomes the front door, and every output leads with the decision. The reframing follows the maintainers' direction — the product most wanted is "look at a lecture and advise whether a JAX upgrade is recommended" — and the measured record agrees: in every evaluation to date (ge_arrow, markov_asset, wald_friedman, and the 2026-08-06 ge_arrow re-run on [skills#10](https://github.com/QuantEcon/skills/issues/10)) the recommendation was decided by the triage-layer instruments — the as-used baseline and what a conversion could reach — and never moved by the scorecard on top. Review mode is unchanged and stays: it is the mode that caught markov_asset's masked build defect, and it applies the day a conversion PR exists.
+
+**Changed**
+
+- `SKILL.md` leads with triage — the no-candidate "should this lecture be converted?" question — behind a "Which mode" router, with review as the second mode. The frontmatter description now opens with the advise use case, so natural-language invocation matches the common question. Review-mode content is unchanged.
+- The scorer's printed output and the report format lead with the verdict. `score.py` prints `VERDICT:` above the weighted total, labels the total "for the record", and the deciding-flip lines name the verdict they flip to before the recomputed number (previously `⇒ total 2.30, …`, which two careful readers in a row took as the headline — [skills#14, finding 6](https://github.com/QuantEcon/skills/issues/14)). The report's TL;DR opens with the full verdict and carries the score alongside as candidate quality for the record; the dimension table gains a verdict row so it still carries the decision when quoted on its own.
+- `README.md` puts triage first throughout — the mode table, the invocation examples, and the mode sections — and states that triage builds no candidate: it measures the lecture as it stands and bounds what a conversion could deliver.
+- Triage now names its canonical decision criteria: the manual's JAX style page ([when to use JAX, when not to](https://manual.quantecon.org/styleguide/jax.html), including *Converting from Numba § Decide first*), cited rather than restated. The skill's four checks are framed as the measurement layer that tests whether those criteria hold for a given lecture — "a real bottleneck" is a claim the as-used baseline and pattern match establish or refute, while the page's "teaching JAX itself" criterion is editorial and stays a maintainer call.
+- The triage decision rule no longer re-derives numbers from the rubric weights: it states the conclusion qualitatively and points to `references/EVALUATION_FRAMEWORK.md`, which the skill's own scoring step already said was the only place weights live. Triage's outcome vocabulary is standardized on **convert / don't-convert** in both `SKILL.md` and `README.md`. (Caught by Copilot's review of the 0.4.0 PR.)
+
+Nothing in the rubric, weights, gates, or scorecard JSON changed: the regression anchors (2.85 / 2.25) and the fixtures reproduce unchanged.
+
 ## 0.3.2 — 2026-08-03
 
 **Fixed**

benchmark/README.md

@@ -2,46 +2,32 @@
 
 Evaluation tooling for QuantEcon lecture code rewrites — the question it answers is never "is JAX faster?" but **"does this implementation earn its place in this lecture?"** Lectures are teaching materials first and programs second; the plugin's rubric weights readability (0.25) above efficiency (0.15) on purpose.
 
-One skill, two modes:
+One skill, two modes — triage is the front door, since most questions arrive before any code is written:
 
 | Mode | Question | Needs | Produces |
 |---|---|---|---|
+| **Triage** | Is this lecture worth converting at all? | the existing lecture only | A recommendation with the binding constraint named |
 | **Review** | Did this conversion PR improve the lecture? | baseline + candidate implementations | A scored report with a merge recommendation |
-| **Triage** | Is this lecture worth converting at all? | the existing lecture only | A predicted verdict band with the binding constraint named |
 
 Status: operational for workspace runs since v0.3.0, which wired the skill to the scoring engine — what shipped in which release is in [CHANGELOG.md](CHANGELOG.md), and remaining plan items are in [skills#4](https://github.com/QuantEcon/skills/issues/4). The system was developed and validated by [@xuanguang-li](https://github.com/xuanguang-li) on [lecture-python.myst#717](https://github.com/QuantEcon/lecture-python.myst/pull/717) and [#654](https://github.com/QuantEcon/lecture-python.myst/pull/654).
 
 ## Using the skill
 
 ```
-/benchmark:review-acceleration <PR number or baseline..candidate refs>   # review mode
 /benchmark:review-acceleration should we convert <lecture>?              # triage mode
+/benchmark:review-acceleration <PR number or baseline..candidate refs>   # review mode
 ```
 
-### Review mode — what you get
-
-The skill follows the measure → record-evidence → score contract ([scripts/README.md](scripts/README.md)): it extracts both implementations verbatim from the lecture's code cells, adapts the measurement templates, runs them, fills `evidence.json` with cited answers, and lets the engine compute the verdict — **no score is ever typed by hand**. The session shows the engine's derivation table (every score with the measured number and threshold band that produced it), and the final report follows the worked examples' format:
-
-1. **TL;DR** — weighted score, verdict band, the decisive facts in one paragraph
-2. **Dimension table** — weight / score / weighted contribution / one-line driver each
-3. **What changed** — before/after implementation shape
-4. **Evidence by dimension** — `max|Δ|` in both dtype regimes, prerequisite-concept and docstring deltas, the as-used vs warm timing table, crossover-n, recompile audit
-5. **Recommendation** — a must-fix list where each item is tagged with the dimension it lifts, plus where the score lands after fixes
-
-See [references/examples/ge_arrow/ge_arrow_REPORT.md](references/examples/ge_arrow/ge_arrow_REPORT.md) (2.85/5, no-conversion; candidate band mixed/wash) and [references/examples/markov_asset/markov_asset_REPORT.md](references/examples/markov_asset/markov_asset_REPORT.md) (2.25/5, no-conversion + gated net regression) for complete real reports. Verdict bands, the v2 verdict gates / no-conversion rule / sensitivity stamp, weights, and scoring anchors: [references/EVALUATION_FRAMEWORK.md](references/EVALUATION_FRAMEWORK.md) §1–2.
-
-**The one rule to remember:** warm-only speedups are never the headline. The ge_arrow case measured 1.4–4.8× faster warm and **45× slower as-used** — the as-used number (fresh process, actual problem sizes, compile time included) decides the efficiency score.
-
 ### Triage mode — before any code is written
 
-Four checks, using only the existing lecture:
+The decision criteria are canonical in the manual's JAX style page — [when to use JAX, when not to](https://manual.quantecon.org/styleguide/jax.html) — and the advice cites them; the skill's contribution is the measurement layer that tests whether they hold. Four checks, using only the existing lecture — no candidate is built:
 
 1. **Baseline as-used total** — replay the lecture's real call sequence (the NumPy half of an `as_used_total.py` template). This bounds the entire possible win: a lecture whose compute totals 30ms has nothing to give.
 2. **Workload-pattern match** — against the two calibrated poles: **aiyagari-shaped** (large fixed-shape arrays, many re-solves, stable static args → measured ~24× as-used win) vs **ge_arrow-shaped** (tiny models, fresh static args per call → measured ~45× as-used loss).
 3. **Crossover comparison** — the lecture's problem sizes vs the warm crossover-n from the scaling data.
 4. **Readability-cost forecast** — which concepts the conversion would force on readers (static args, `lax` carries, checkify, the float32/x64 distinction), against the prerequisite-concept bands.
 
-Then the decision rule that falls out of the rubric weights: efficiency (0.15) can gain at most +0.30 weighted (band 3→5), while readability (0.25) losing two bands costs −0.50 — **a conversion that costs meaningful readability cannot break even on speed alone**; it must also win on logic & design and ergonomics, and those structural wins are usually achievable in plain NumPy.
+Then the decision rule ([EVALUATION_FRAMEWORK.md §1](references/EVALUATION_FRAMEWORK.md)): the rubric weights readability above efficiency on purpose, so **a conversion that costs meaningful readability cannot break even on speed alone** — the weight algebra behind that lives in the framework, not here; it must also win on logic & design and ergonomics, and those structural wins are usually achievable in plain NumPy.
 
 **Validation (2026-07-21):** triage applied blind (baseline-side data only) to the three known cases reproduces every known verdict:
 
@@ -55,6 +41,20 @@ The baseline totals above are the **triage-time** measurements taken on 2026-07-
 
 Scope limit, confirmed by the same test: triage predicts whether the prize is worth pursuing — it cannot predict conversion-quality outcomes (markov_asset's masked `err.throw()` defect was a property of the PR, invisible to triage). Note also that this validation is **in-sample** — the three cases are the ones the thresholds were calibrated on; out-of-sample validation accumulates as fresh lectures are triaged.
 
+### Review mode — what you get
+
+The skill follows the measure → record-evidence → score contract ([scripts/README.md](scripts/README.md)): it extracts both implementations verbatim from the lecture's code cells, adapts the measurement templates, runs them, fills `evidence.json` with cited answers, and lets the engine compute the verdict — **no score is ever typed by hand**. The session shows the engine's derivation table (every score with the measured number and threshold band that produced it), and the final report follows the worked examples' format, leading with the decision:
+
+1. **TL;DR** — the *full* verdict first (gate / no-conversion / sensitivity included), the decisive facts, and the weighted score alongside as candidate quality for the record
+2. **Dimension table** — a verdict row, then weight / score / weighted contribution / one-line driver each, so the table still carries the decision when quoted on its own
+3. **What changed** — before/after implementation shape
+4. **Evidence by dimension** — `max|Δ|` in both dtype regimes, prerequisite-concept and docstring deltas, the as-used vs warm timing table, crossover-n, recompile audit
+5. **Recommendation** — a must-fix list where each item is tagged with the dimension it lifts, plus where the score lands after fixes
+
+See [references/examples/ge_arrow/ge_arrow_REPORT.md](references/examples/ge_arrow/ge_arrow_REPORT.md) (2.85/5, no-conversion; candidate band mixed/wash) and [references/examples/markov_asset/markov_asset_REPORT.md](references/examples/markov_asset/markov_asset_REPORT.md) (2.25/5, no-conversion + gated net regression) for complete real reports. Verdict bands, the v2 verdict gates / no-conversion rule / sensitivity stamp, weights, and scoring anchors: [references/EVALUATION_FRAMEWORK.md](references/EVALUATION_FRAMEWORK.md) §1–2.
+
+**The one rule to remember:** warm-only speedups are never the headline. The ge_arrow case measured 1.4–4.8× faster warm and **45× slower as-used** — the as-used number (fresh process, actual problem sizes, compile time included) decides the efficiency score.
+
 ## Manual usage (no skill)
 
 The full recipe is in [scripts/README.md](scripts/README.md) ("Evaluating a new lecture"); quickstart from this directory:

benchmark/scripts/scoring/score.py

@@ -127,8 +127,13 @@ def main(lecture_dir):
               f"{r['score']:>3d} {r['weighted']:>5.2f}")
         print(f"     └ {r['reason']}")
     print("-" * 78)
-    print(f"{'WEIGHTED TOTAL':34s} {'':11s} {'':>4s} {'':>3s} {result['total']:>5.2f}")
+    # The verdict prints before the number: the two are different layers (the
+    # total measures polish, the verdict carries the decision), and a reader
+    # who takes away one line must take away the recommendation. Both the
+    # system's designer and its sponsor read the total as the headline when it
+    # came first — skills#14, finding 6.
     print(f"VERDICT: {result['verdict']}")
+    print(f"{'WEIGHTED TOTAL (for the record)':34s} {'':11s} {'':>4s} {'':>3s} {result['total']:>5.2f}")
     skipped = sens["perturbations_skipped"]
     print(f"SENSITIVITY: {sens['stamp']} "
           f"({sens['perturbations_tested']} single-input perturbations scored"
@@ -138,7 +143,8 @@ def main(lecture_dir):
         print(f"     └ {sens['stamp_note']}")
     for fl in sens["deciding_flips"]:
         print(f"     └ {fl['input']}: {fl['from']} → {fl['to']} "
-              f"⇒ total {fl['total']:.2f}, {fl['outcome']}")
+              f"⇒ verdict {fl['outcome']} "
+              f"(recomputed total, for the record: {fl['total']:.2f})")
 
     out = {
         "lecture": lecture,