benchmark: triage first — advise before code is written; every output leads with the decision (0.4.0)

[mmcky]

Reframes the benchmark skill around the question the org actually keeps asking, without removing anything: triage — "should this lecture be converted at all?" — becomes the front door, and review mode stays intact for the day a conversion PR exists again.

Why

Two grounds, both measured rather than aesthetic.

The demand is on triage's side. Every conversion-PR route the acceptance test was waiting on closed in one sweep: on 2026-08-05 QuantEcon/lecture-python.myst#657, #624 and #644 all merged with their JAX content dropped (style fixes only landed), after #654 was closed with the lecture staying in NumPy. There are currently zero open conversion PRs, while Table C alone lists eight candidate lectures with no PR — the prospective question is the one being asked, and it was the skill's second-billed mode.

The record says the triage layer decides. In every evaluation to date — ge_arrow, markov_asset, wald_friedman, and the 2026-08-06 ge_arrow re-run on #10 — the recommendation was determined by the as-used baseline and what a conversion could reach; the scorecard on top never flipped an outcome. Meanwhile the number-first presentation demonstrably misled: two careful readers in a row took the total for the headline (#14, finding 6).

What changed

• SKILL.md leads with triage behind a short "Which mode" router; the frontmatter description opens with the advise use case so natural-language invocation matches the common question; triage states explicitly that no candidate is built. Review-mode content is unchanged, one step excepted (below).
• Triage cites its canonical decision criteria — the manual's JAX style page (when to use JAX, when not to, including Converting from Numba § Decide first) — rather than restating them, and frames its four checks as the measurement layer that tests whether those criteria hold for a given lecture. "A real bottleneck" is a claim the as-used baseline and pattern match establish or refute; the page's "teaching JAX itself" criterion is editorial and stays a maintainer call.
• The triage decision rule is stated qualitatively, with the weight algebra left to the framework (from Copilot's review of this PR): SKILL.md and README.md had both re-derived numbers from the rubric weights while the skill's own scoring step says weights live only in references/EVALUATION_FRAMEWORK.md. Both now state the conclusion and point there, and triage's outcome vocabulary is standardized on convert / don't-convert, replacing the review-mode "predicted verdict band".
• Recommendation-first output, everywhere. score.py prints VERDICT: above the weighted total, labels the total "for the record", and the deciding-flip lines name the verdict they flip to before the recomputed number. The report format's TL;DR opens with the full verdict with the score carried alongside as candidate quality for the record, and the dimension table gains a verdict row so it still carries the decision when quoted on its own. This lands TESTING: benchmark triage run on wald_friedman — six plugin gaps, and a floor that measures the wrong quantity #14's finding 6.
• README.md puts triage first throughout (mode table, invocation examples, sections); the tutorial's quoted engine output follows the new format.
• benchmark 0.3.2 → 0.4.0 with a changelog entry (materially different emphasis → minor bump).

What deliberately did not change

The rubric, weights, gates, verdict bands, sensitivity machinery, and scorecard JSON are untouched — CI's regeneration check passes with byte-identical scorecards, and the anchors reproduce 2.85 / no-conversion / fragile and 2.25 / no-conversion / robust-at-floor. Review mode caught markov_asset's masked build defect and remains the mode that answers "should this PR merge?" when one arrives.

Remaining follow-up (not this PR): QuantEcon.manual#104 — distilling what the validation program measured (the 1 s floor, the addressable-share lesson from wald_friedman, the readability break-even) into the manual's JAX page as the rubric's companion, so the canonical criteria absorb the program's empirical findings. Kenko's review is requested in the comments below and on #10.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR reframes the benchmark plugin’s review-acceleration skill to be triage-first and makes the scorer/reporting output recommendation-first (verdict printed before the weighted total), while bumping the plugin version to 0.4.0 and updating docs accordingly.

Changes:

• Updated score.py output ordering and sensitivity “deciding flips” lines to lead with the verdict and treat the weighted total as “for the record”.
• Reordered and revised skill/docs/README content so triage is presented as the default entry point and outputs emphasize the decision first.
• Released as benchmark v0.4.0 with matching manifest/version updates and changelog entry.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
docs/tutorial-run-an-evaluation.md	Updates the tutorial’s quoted scorer output and explains verdict-first semantics.
benchmark/skills/review-acceleration/SKILL.md	Reorders the skill doc to triage-first and reinforces “decision-first” reporting.
benchmark/scripts/scoring/score.py	Prints VERDICT: before the weighted total; updates deciding-flip display text.
benchmark/README.md	Reorders mode presentation to put triage first; updates review-mode description to match decision-first outputs.
benchmark/CHANGELOG.md	Adds 0.4.0 release notes describing the triage-first reframing and verdict-first output.
benchmark/.claude-plugin/plugin.json	Bumps plugin version to 0.4.0.
.claude-plugin/marketplace.json	Bumps marketplace entry for benchmark to 0.4.0.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mmcky]

@xuanguang-li — requesting your review on this PR, as the designer of the evaluation system it reorders. Context is in the description above and on #10; the short version: triage ("should this lecture be converted at all?") becomes the skill's front door, and every output now leads with the recommendation rather than the number — your two comments on #10 are the documented evidence for that second change.

What we would most value your judgement on, in order:

1. Does triage-first lose anything you consider load-bearing? The reordering demotes nothing — review mode and the rubric are untouched, and CI reproduces both anchors byte-identically — but you know the system's intent better than anyone.
2. Is the verdict-first presentation right? score.py now prints VERDICT: above the weighted total and labels the total "for the record"; the deciding-flip lines name the verdict they flip to before the recomputed number. This is the fix for the two-layer confusion you hit (and reported) on TESTING: review + expert-parallel protocol for the benchmark skill (@xuanguang-li) #10.
3. The triage decision rule is now qualitative — "readability outranks efficiency on purpose, so a conversion that costs meaningful readability cannot break even on speed alone" — with the weight algebra left to EVALUATION_FRAMEWORK.md. Confirm the distillation is faithful to the rule you derived.

No urgency on the rubric-confirmation items from the #10 checklist — this PR is the priority ask, and none of it forecloses those.

[xuanguang-li]

Thanks @mmcky. This summary is helpful! I'll look into it tonight.

[xuanguang-li]

Hi @mmcky,

Thanks for this update.

1. I think it's fine. I confirmed that this PR doesn't touch the pipeline that produces the weighted score. It only changes the way the result is presented. However, with start here in the title of triage mode, Claude automatically did a triage without asking me which one it should choose. Following is Claude's reasoning:

My reasoning for triage was that the invocation was a bare ge_arrow with no PR number or refs, and the skill calls triage the front door — "start here." That justifies triage when you don't know yet.

2. Yes, the report shows the verdict first after I run the skill now, and the explanation of the sensitivity test is clearer.

3. Since this rule is used for triage mode, I think it relates to Finding 2 in issue PLAN: evaluation rubric v2 — enforce couplings, verdict vocabulary, instrument fixes (from the three-way design review) #7, which pointed out the lack of a scaffold for the triage mode. If we decide not to build a scaffold for triage mode, this qualitative rule is fine. But if we decide to move forward, this rule will need to be revised.

[mmcky]

Thanks @xuanguang-li — appreciated, especially the check that the scoring pipeline is untouched.

On 1: that is the intended default — a bare lecture with no PR or refs reads as "should this be converted at all?", which is triage's question, so it goes first. But you're right that the choice was made silently; having the skill state which mode it picked and why (and ask when it's genuinely ambiguous) is a fair refinement, and worth a small follow-up rather than holding this PR.

On 3: agreed — the qualitative rule is written for the current no-scaffold triage, so it stands or gets revised together with Finding 2 in #7 when that decision is made.

I'll merge this as-is and tag benchmark 0.4.0.