fix(grade): match transcript_check tool patterns by descriptor role

[slowdini]

Closes #308.

The problem

tool_invocation_matches applied its authored regex only to the native invocation rendering. A frozen pattern such as Bash|Read matches Claude Code's tool names but not Codex's command_execution, so a Codex fleet run scored 0/10 on every mechanical transcript assertion despite direct evidence of the behavior in all 80 runs. The same behavioral assertion reported different results purely because a harness picked a different native tool name.

The descriptor vocabulary was already the portability seam everywhere else — sandbox::policy and detect_stray_writes classify against all_tool_vocabulary() so "no harness's tool naming is hardcoded here", and must_precede: first_write was already role-aware. tool_invocation_matches was the one holdout.

The change

Tool names now match portably, in two stages:

1. The regex runs against the native rendering, exactly as the run recorded it.
2. On a miss, the run's own descriptor supplies the role its tool name belongs to (write/patch/shell/read — validated disjoint per descriptor), and the registry-wide union supplies every portable spelling of that role. Only the name is substituted.

Nothing names a harness and no alias list is hard-coded, so a BYOH descriptor opts in through its [tools] table alone — declaring shell = ["cool_exec"] is the whole opt-in, with no need to cross-list other harnesses' names. No shipped descriptor's [tools] changed.

Preserved as-is: argument regexes (arguments are carried through the substitution verbatim), assistant_message_matches, must_precede semantics (ordering still classifies against the run's own vocabulary only — the union could call another harness's name a write), and native-name evidence wording. A tool the run's descriptor declares in no role is matched by its native name alone; nothing is invented for it.

Before / after

One authored assertion, "pattern": "Bash|Read", against a Codex run whose only invocation is command_execution:

before  no candidate matched /Bash|Read/ across 1 invocation(s)
after   matched ordinal 0 via shell alias 'Bash': command_execution {"command":"bun test"}

Evidence keeps the two kinds of match apart: a native match reads matched ordinal 0: Bash {"command":"ls"} with no alias mentioned, and an alias match reports the invocation the harness actually recorded alongside the alias and role that matched it. A miss now names the roles that were expanded:

no candidate matched /Bash|Read/ across 12 invocation(s) (native names plus write/shell role aliases)

Note on granularity

Aliasing is role-wide, so within a role names are interchangeable — on Claude Code a Glob call now satisfies a Read pattern, since both are read tools. That is the ticket's stated contract and is inherent to any role-based scheme. The shipped guide documents it and points at the workaround: key the pattern off arguments when tools inside one role must be told apart.

Scope audit

Every surface that compares an authored pattern or a tool name was checked. grade, grade --finalize, and ingest all funnel through one finalize() call site, so the fix covers all three. judge-tasks, compare, aggregate, judge-evidence.md, and validate are unaffected; the guard and stray-write audit were already union-based.

Tests

• 11 unit tests in the new src/pipeline/grade/transcript_check/alias_tests.rs: alias matches, native precedence, undeclared names, cross-role isolation, both ordering directions, argument regexes, empty vocabulary, and the roles-expanded miss message.
• ToolRole / ToolVocabulary unit tests in src/adapters/harness.rs.
• End-to-end regression guard in tests/cli/grade.rs: one Bash|Read assertion graded through grade --finalize passes on Codex, OpenCode, Cline and Claude Code. OpenCode and Cline declare only bash / run_commands, so they fail without the union — this is the case that pins the design.
• BYOH coverage in tests/run/byoh.rs: a descriptor declaring shell = ["zap_exec"] — a name no other descriptor knows — grades a Bash|Read assertion through [tools] alone.
• Legacy fallback: an iteration whose conditions.json names no harness matches native names only, with byte-identical pre-change evidence.

Documentation

• docs/guides/judging.md — new "Portable tool patterns" section (shipped as eval-magic docs judging) with the two-stage rule, a worked example across all four built-ins, the evidence formats, and the role-granularity consequence.
• docs/guides/byoh.md — what [tools] buys a custom harness, and that cross-listing is neither needed nor wanted.
• schema/evals.schema.json — transcriptCheck.pattern description.
• --help for grade and ingest; the [tools] comment in harnesses/template.toml.
• Internal notes: codex-notes.md, cline-notes.md, progressive-enhancements.md.

Verification

cargo test                              967 + 220 + 209 passing, 0 failures
cargo fmt --check                       clean
cargo clippy --all-targets -- -D warnings   clean
cargo run -- docs judging               new section renders

transcript_check.rs crossed 500 lines, so the new tests were split into a transcript_check/alias_tests.rs sibling per the repo's documented extraction pattern; the remaining file is 366 lines of production code plus the pre-existing 165-line inline test module.

🤖 Generated with Claude Code

https://claude.ai/code/session_018CGVCheSsLsP2jdHhtrtER