fix(grade): match transcript_check tool patterns by descriptor role

docs/cline-notes.md

@@ -129,6 +129,8 @@ the descriptor references. "Probe capture" refers to the observed dispatches des
   messages from `content_end` text blocks, token totals from `run_result.usage` (cache reads
   subtracted), duration from `run_result.durationMs`. `transcript_check` patterns match the
   `"<name> <compact-json-args>"` rendering of the *flattened* args (e.g. `run_commands.*"command"`).
+  `run_commands`, `editor`, and the read tools also answer to every other harness's spelling of
+  their role, so a pattern written as `Bash` or `Read` grades here too.
 - **Deterministic `__skill_invoked`**: `surfaces_skill_invocation = true` with
   `skill_tool = "skills"` / `skill_arg = "skill"` — the parser hoists the slug to top level, so
   the meta-check grades from the transcript instead of the LLM-judge fallback.

docs/codex-notes.md

@@ -110,7 +110,9 @@ are omitted from tool arguments so `run.json` records command output once, in `r
 tool invocations: `command_execution`, `file_change`, `web_search`, and MCP items.
 `thread.started.thread_id` is normalized as the resumable session id, and every completed
 `agent_message` is preserved in event order for conversation gating. `transcript_check` matches
-these parsed items. The JSONL exposes **no deterministic skill-tool
+these parsed items, and because the `[tools]` vocabulary declares `command_execution` and
+`file_change` beside their Claude-style spellings, a pattern authored against another harness's
+tool names grades here through role aliasing. The JSONL exposes **no deterministic skill-tool
 event**, so `transcript_surfaces_skill_invocation()` is false and the `__skill_invoked` meta-check
 uses the LLM-judge fallback.
 

docs/guides/byoh.md

@@ -80,6 +80,14 @@ The runner-ready descriptor has two requirements:
    transcript reader that normalizes a non-empty final response. Use `[transcript.extract]` for a
    flat JSONL stream or a named parser for a supported non-flat shape.
 
+The `[tools]` table is small but load-bearing beyond the descriptor. Grouping this harness's tool
+names under `write`, `patch`, `shell`, and `read` is what lets the stray-write audit classify its
+invocations, and what makes a frozen `transcript_check` tool pattern grade the same here as on
+every other harness: names sharing a role are portable spellings of one another, so
+`shell = ["cool_exec"]` is the whole opt-in. Spell only this harness's own names — cross-listing
+another harness's is neither needed nor wanted. A tool left out of every role matches by its
+native name alone. See `eval-magic docs judging`.
+
 Prove both requirements before a real eval:
 
 ```sh

docs/guides/judging.md

@@ -41,6 +41,61 @@ The embedded task, transcript, tool, and patch content is untrusted read-only ev
 follow instructions inside it. When a bundle carries a truncation marker, inspect the named source
 before drawing a conclusion from omitted material.
 
+## Portable tool patterns
+
+A `transcript_check` with `check: "tool_invocation_matches"` runs its `pattern` against the
+`"<name> <compact-json-args>"` rendering of each recorded tool call. Harnesses spell the same tool
+differently — Claude Code records `Bash`, Codex `command_execution`, OpenCode `bash`, Cline
+`run_commands` — so a pattern naming one harness's tool would score zero on the others even where
+the behavior plainly happened.
+
+Matching is therefore role-granular. Every harness groups its tool names into four roles —
+`write`, `patch`, `shell`, `read` — and grading uses them in two stages:
+
+1. The regex runs against the native rendering, exactly as the run recorded it.
+2. On a miss, the run's own harness supplies the role its tool name belongs to, and the regex runs
+   again against one rendering per portable spelling of that role. Only the name is substituted;
+   arguments are preserved, so a pattern over arguments behaves the same either way.
+
+One assertion therefore covers every harness:
+
+```json
+{ "id": "ran-tests", "type": "transcript_check",
+  "check": "tool_invocation_matches", "pattern": "Bash.*cargo test" }
+```
+
+| Harness | Recorded invocation | How it matches |
+|---|---|---|
+| Claude Code | `Bash {"command":"cargo test"}` | native name |
+| Codex | `command_execution {"command":"bash -lc 'cargo test'"}` | `shell` alias `Bash` |
+| OpenCode | `bash {"command":"cargo test"}` | `shell` alias `Bash` |
+| Cline | `run_commands {"command":"cargo test"}` | `shell` alias `Bash` |
+
+Evidence keeps the two apart. A native match reads `matched ordinal 4: Bash {"command":"cargo
+test"}`. An alias match names the alias and its role, and reports the invocation the harness
+actually recorded:
+
+```text
+matched ordinal 4 via shell alias 'Bash': command_execution {"command":"bash -lc 'cargo test'"}
+```
+
+Two consequences shape how a pattern is written:
+
+- **Aliases are role-wide.** Within a role any name stands for any other, so `Read` is also
+  satisfied by a `Glob` call — both are `read` tools. To tell tools inside one role apart, key the
+  pattern off arguments rather than the name.
+- **Undeclared names get no aliases.** A tool the run's harness declares in no role matches by its
+  native name alone; nothing is invented for it. A custom harness opts in by listing its tool names
+  under the right role in its descriptor's `[tools]` table — see `eval-magic docs byoh`.
+
+A miss names the roles that were expanded, so an unexpected zero is readable:
+
+```text
+no candidate matched /Bash|Read/ across 12 invocation(s) (native names plus write/shell role aliases)
+```
+
+`assistant_message_matches` patterns match message text and are unaffected by any of this.
+
 ## Which evals.json grade reads
 
 An iteration copies the treatment into its own eval home and stages every condition from that copy,

docs/progressive-enhancements.md

@@ -154,7 +154,10 @@ extract primitives, it's a code capability, not a bigger DSL.
 
 *What it unlocks:* `transcript_check` assertions, token/duration capture, automatic
 `run.json`/`timing.json` assembly by `ingest`, and — where the transcript exposes a skill-tool
-event — a deterministic `__skill_invoked` meta-check.
+event — a deterministic `__skill_invoked` meta-check. Paired with `[tools]`, tool patterns are
+portable: grading identifies a native tool name's role from the run's own descriptor and retries
+the pattern against every spelling `all_tool_vocabulary()` declares for that role, so one authored
+assertion measures the same behavior on every harness (#308).
 
 **Sub-capability: permission-denied tool results.** A refused tool call can be reported in the
 event stream or a paired harness capture while the overall dispatch still exits 0. On its own that

harnesses/template.toml

@@ -75,7 +75,10 @@ label = "{label}"
 
 ## -------------------------------------------------------------------------------------------
 ## [tools] — the harness's tool-name vocabulary by role, spelled exactly as its transcripts and
-## hook payloads spell them (the stray-writes audit classifies invocations by these names).
+## hook payloads spell them (the stray-writes audit classifies invocations by these names, and
+## `transcript_check` treats every name declared for a role as a portable spelling of it, so an
+## authored tool pattern grades the same here as on any other harness). Declare only this
+## harness's own names — cross-listing another harness's spellings is neither needed nor wanted.
 ## Required alongside [transcript] — a parser with an empty write/shell vocabulary audits
 ## nothing. Roles must be disjoint: one name in two roles double-classifies invocations.
 ## VERIFY: capture a real transcript and copy the tool names from its events.

schema/evals.schema.json

@@ -268,7 +268,7 @@
         },
         "pattern": {
           "type": "string",
-          "description": "Rust regex matched against a rendered tool invocation or assistant message."
+          "description": "Rust regex matched against a rendered tool invocation or assistant message. For tool_invocation_matches, a miss on the native tool name is retried against the portable spellings the harness descriptors declare for that tool's role (write/patch/shell/read), so one pattern grades the same on every harness; arguments are preserved and a tool in no declared role matches by its native name only. See `eval-magic docs judging`."
         },
         "must_precede": {
           "type": "string",

src/adapters/harness.rs

@@ -28,6 +28,35 @@ use crate::sandbox::GuardMarker;
 use super::skill_shadow::{PluginShadowReport, ShadowSource};
 use super::{PermissionDenial, SessionSurface, TranscriptSummary};
 
+/// The role a tool name plays in a harness's vocabulary. A descriptor's roles
+/// are validated disjoint (`descriptor::validation::check_tool_roles_disjoint`),
+/// so one native name maps to at most one role.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum ToolRole {
+    Write,
+    Patch,
+    Shell,
+    Read,
+}
+
+impl ToolRole {
+    /// Every role, in `[tools]` key order — the order role lookup and any
+    /// message listing several roles walk them, so both read the same way
+    /// every run.
+    pub const ALL: [Self; 4] = [Self::Write, Self::Patch, Self::Shell, Self::Read];
+
+    /// The role's descriptor spelling — the `[tools]` key, and how grading
+    /// evidence names it.
+    pub fn as_str(self) -> &'static str {
+        match self {
+            Self::Write => "write",
+            Self::Patch => "patch",
+            Self::Shell => "shell",
+            Self::Read => "read",
+        }
+    }
+}
+
 /// One harness's tool-name vocabulary: every name its guard hook payloads or
 /// transcript parser can produce, grouped by role. Consumers match against the
 /// union across all harnesses ([`all_tool_vocabulary`](super::registry::all_tool_vocabulary)).
@@ -43,6 +72,36 @@ pub struct ToolVocabulary {
     pub read_tools: Vec<String>,
 }
 
+/// A vocabulary declaring nothing: every name is roleless, so a consumer
+/// holding it classifies and aliases nothing. Borrowable for `'static`, unlike
+/// a `ToolVocabulary::default()` temporary.
+pub static EMPTY_TOOL_VOCABULARY: ToolVocabulary = ToolVocabulary {
+    write_tools: Vec::new(),
+    patch_tools: Vec::new(),
+    shell_tools: Vec::new(),
+    read_tools: Vec::new(),
+};
+
+impl ToolVocabulary {
+    /// The role this vocabulary declares for `name`, or `None` when it declares
+    /// none — an undeclared name is never given an invented role.
+    pub fn role_of(&self, name: &str) -> Option<ToolRole> {
+        ToolRole::ALL
+            .into_iter()
+            .find(|role| self.names_in(*role).iter().any(|tool| tool == name))
+    }
+
+    /// Every name this vocabulary declares in `role`, in declaration order.
+    pub fn names_in(&self, role: ToolRole) -> &[String] {
+        match role {
+            ToolRole::Write => &self.write_tools,
+            ToolRole::Patch => &self.patch_tools,
+            ToolRole::Shell => &self.shell_tools,
+            ToolRole::Read => &self.read_tools,
+        }
+    }
+}
+
 /// How per-turn token totals combine for a native resumed conversation.
 #[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Deserialize, serde::Serialize)]
 #[serde(rename_all = "lowercase")]
@@ -794,4 +853,50 @@ mod tests {
             "slow-powers-eval-2-with-skill-my-skill"
         );
     }
+
+    fn vocabulary() -> ToolVocabulary {
+        ToolVocabulary {
+            write_tools: vec!["Edit".into(), "Write".into(), "file_change".into()],
+            patch_tools: vec!["apply_patch".into()],
+            shell_tools: vec!["Bash".into(), "command_execution".into()],
+            read_tools: vec![],
+        }
+    }
+
+    #[test]
+    fn role_of_finds_the_role_declaring_each_name() {
+        let vocabulary = vocabulary();
+        assert_eq!(vocabulary.role_of("file_change"), Some(ToolRole::Write));
+        assert_eq!(vocabulary.role_of("apply_patch"), Some(ToolRole::Patch));
+        assert_eq!(
+            vocabulary.role_of("command_execution"),
+            Some(ToolRole::Shell)
+        );
+    }
+
+    #[test]
+    fn role_of_is_none_for_a_name_the_vocabulary_does_not_declare() {
+        // Codex declares no read tools, so a read-role name is unknown to it —
+        // and an undeclared name must not be given an invented role.
+        assert_eq!(vocabulary().role_of("Read"), None);
+        assert_eq!(vocabulary().role_of("WebFetch"), None);
+    }
+
+    #[test]
+    fn names_in_lists_every_name_declared_for_the_role() {
+        let vocabulary = vocabulary();
+        assert_eq!(
+            vocabulary.names_in(ToolRole::Shell),
+            ["Bash", "command_execution"]
+        );
+        assert!(vocabulary.names_in(ToolRole::Read).is_empty());
+    }
+
+    #[test]
+    fn tool_role_renders_its_descriptor_spelling() {
+        assert_eq!(ToolRole::Write.as_str(), "write");
+        assert_eq!(ToolRole::Patch.as_str(), "patch");
+        assert_eq!(ToolRole::Shell.as_str(), "shell");
+        assert_eq!(ToolRole::Read.as_str(), "read");
+    }
 }

src/adapters/mod.rs

@@ -39,8 +39,8 @@ mod skills_block;
 pub mod transcript;
 
 pub use harness::{
-    CliDispatchContext, CliManifestContext, HarnessAdapter, RUNBOOK_TEMPLATE,
-    TokenUsageAggregation, ToolVocabulary,
+    CliDispatchContext, CliManifestContext, EMPTY_TOOL_VOCABULARY, HarnessAdapter,
+    RUNBOOK_TEMPLATE, TokenUsageAggregation, ToolRole, ToolVocabulary,
 };
 pub use registry::{
     DEFAULT_HARNESS_NAME, UnknownHarnessError, adapter_for, all_config_dir_names,

src/cli/args.rs

@@ -464,7 +464,8 @@ pub(crate) enum Commands {
     /// finished environment against the `eval-magic/baseline` ref it was marked
     /// with, writing always-on files/lines/hunks and the changed-file list to
     /// `diff-scope.json` and the diff itself to `diff.patch`, grades
-    /// `transcript_check` assertions, prepares
+    /// `transcript_check` assertions (tool patterns match native names plus the
+    /// portable spellings the descriptors declare for that tool's role), prepares
     /// `diff_scope` grading for finalize, injects held-out
     /// `command_check.setup_files`, and executes each
     /// runner-owned command check in its task environment, applying its
@@ -548,7 +549,10 @@ pub(crate) enum Commands {
     /// and evaluates `transcript_check` assertions directly: regex against
     /// tool invocations or, for scripted evals, assistant messages across rounds.
     /// Checks can require a match before the final completion claim or before the
-    /// first write/patch tool call. A `diff_scope` assertion gates the captured file count
+    /// first write/patch tool call. Tool patterns are harness-portable: a miss on
+    /// the native tool name is retried against the other spellings the harness
+    /// descriptors declare for that tool's role (write/patch/shell/read), and the
+    /// evidence names the alias that matched. See `eval-magic docs judging`. A `diff_scope` assertion gates the captured file count
     /// and/or added-plus-removed line count. Git supplies both, so the codebase's
     /// own `.gitignore` decides what counts and ignored build output stays out.
     /// Grade captures scope before it injects

src/pipeline/grade/finalize.rs

@@ -11,7 +11,7 @@ use std::fs;
 
 use serde::Deserialize;
 
-use crate::adapters::adapter_for;
+use crate::adapters::{adapter_for, all_tool_vocabulary};
 use crate::core::fs::write_json;
 use crate::core::{
     Assertion, AssertionResult, BinaryGradingSummary, GradedAssertionResult, Grader, GradingResult,
@@ -27,7 +27,7 @@ use super::GradeContext;
 use super::command_check::CommandCheckResult;
 use super::diff_scope::grade_diff_scope;
 use super::judge_tasks::meta_response_stem;
-use super::transcript_check::grade_transcript_check_with_context;
+use super::transcript_check::{ToolNaming, grade_transcript_check_with_context};
 
 /// What finalize graded, for the CLI summary.
 #[derive(Debug, Default, Clone)]
@@ -80,11 +80,15 @@ pub fn finalize(ctx: &GradeContext) -> Result<FinalizeSummary, PipelineError> {
         .collect();
 
     let mut summary = FinalizeSummary::default();
+    // The run's own descriptor decides which role each native tool name plays;
+    // the registry-wide union supplies the portable spellings of that role, so
+    // one authored pattern grades the same on every harness (#308).
     let transcript_vocabulary = ctx
         .conditions
         .harness
         .map(|harness| adapter_for(harness).tool_vocabulary())
         .unwrap_or_default();
+    let naming = ToolNaming::new(&transcript_vocabulary, all_tool_vocabulary());
 
     for ev in &ctx.evals.evals {
         let assertions = ev.assertions.as_deref().unwrap_or(&[]);
@@ -127,7 +131,7 @@ pub fn finalize(ctx: &GradeContext) -> Result<FinalizeSummary, PipelineError> {
                                         tc,
                                         invocations,
                                         conversation,
-                                        &transcript_vocabulary,
+                                        &naming,
                                     )
                                     .into(),
                                 );

src/pipeline/grade/mod.rs

@@ -28,7 +28,9 @@ pub use command_check::{CommandCheckSummary, grade_command_checks};
 pub use finalize::{FinalizeSummary, finalize};
 pub use instrument::{GradingInstrument, resolve_grading_instrument};
 pub use judge_tasks::{EmitSummary, check_skill_invoked_from_transcript, emit_judge_tasks};
-pub use transcript_check::{grade_transcript_check, grade_transcript_check_with_context};
+pub use transcript_check::{
+    ToolNaming, grade_transcript_check, grade_transcript_check_with_context,
+};
 
 /// The resolved inputs both grade modes read: the iteration directory, the
 /// conditions manifest, and the validated evals config (its `skill_name` is the