fix(codex): verify staged skill access

docs/codex-notes.md

@@ -112,9 +112,16 @@ tool invocations: `command_execution`, `file_change`, `web_search`, and MCP item
 `agent_message` is preserved in event order for conversation gating. `transcript_check` matches
 these parsed items, and because the `[tools]` vocabulary declares `command_execution` and
 `file_change` beside their Claude-style spellings, a pattern authored against another harness's
-tool names grades here through role aliasing. The JSONL exposes **no deterministic skill-tool
-event**, so `transcript_surfaces_skill_invocation()` is false and the `__skill_invoked` meta-check
-uses the LLM-judge fallback.
+tool names grades here through role aliasing.
+
+The JSONL exposes no dedicated skill-tool event. A bounded Codex CLI 0.152.1 probe instead emitted
+a successful `item.completed` `command_execution` whose `command` contained the exact staged
+`SKILL.md` path as a literal argument to `sed`; the small synthetic event shape is preserved in
+`tests/fixtures/codex/skill-access-0.152.1.jsonl`. The descriptor's
+`[transcript.skill_access]` table therefore names `command_execution`, its command and exit-code
+arguments, and the accepted read-command basenames. `__skill_invoked` passes locally only when one
+of those commands exits zero with the task's exact staged path. A live source, another treatment
+member's path, dynamic or partial path text, and response wording do not count.
 
 Codex token totals use its blended workload metric:
 

docs/guides/judging.md

@@ -173,13 +173,22 @@ Use `run --judge-samples N` to set a campaign-wide default. An assertion's `samp
 precedence over that default. The effective count must be at least one. The framework-injected
 `__skill_invoked` checks are not substantive grading and remain single-shot per treatment member.
 
-For a multi-skill treatment, deterministic transcript grading checks every staged slug separately.
+For a multi-skill treatment, deterministic transcript grading checks every member separately. A
+native skill-tool signature compares its identifier with that member's staged slug. An exact-path
+access signature instead requires a successful declared read command whose literal argument is
+that member's task-specific staged `SKILL.md` path. Failed commands, skill-name text, final-message
+phrasing, the live source path, and another treatment member's staged path do not count. A local
+result has `confidence: 1.0` and emits no meta judge task.
+
 The response files use `__skill_invoked__skill-N.json`, and each meta result names its
-`skill_name`, so partial and complete invocation are distinguishable. Harness descriptors supply
-the tool and argument signature; a harness without deterministic invocation events receives one
-LLM fallback task per member. The suite-level `meta_summary.skill_invoked` value is true when any
-treatment member was invoked. The `benchmark.json` file retains the suite rate and adds per-skill
-counts and rates. A scalar treatment keeps the `__skill_invoked.json` filename and artifact shape.
+`skill_name`, so partial and complete access or invocation are distinguishable. A run with no
+usable deterministic signature—for example, a descriptor that exposes none—receives one clearly
+labeled behavioral-influence fallback task per member. That fallback can estimate whether the
+skill influenced the response, but a pass does not prove native invocation or staged-file access.
+The compatibility field
+`meta_summary.skill_invoked` is true when any treatment member passes its available check. The
+`benchmark.json` file retains the suite rate and adds per-skill counts and rates. A scalar treatment
+keeps the `__skill_invoked.json` filename and artifact shape.
 
 Each sample is a separate judge task and response, but every sample for a run receives the exact
 same bounded `judge-evidence.md`. The agent is not rerun, and eval-magic does not rebuild or expand

docs/progressive-enhancements.md

@@ -190,7 +190,10 @@ or use their documented grading fallback.
 
 *Descriptor fields:* the `[transcript]` table — `events_filename` (gate: an absent table means the
 ingest pipeline never reads a transcript), one primary summary reader, and
-`surfaces_skill_invocation`. The primary reader is either `parser` or the summary outputs under
+`surfaces_skill_invocation`. A deterministic native skill event uses `skill_tool` / `skill_arg`;
+a successful exact-path shell read uses the mutually exclusive `skill_access` table with its tool,
+command argument, exit-code argument, and declared read-command basenames. The primary reader is
+either `parser` or the summary outputs under
 `extract`; validation rejects both, neither, and a surface-only extract. The `extract` sub-table is
 the declarative tier: equality `where` filters, final and ordered assistant-text picks, a session-id
 pick, flat tool-item mapping, token sum/subtract reduction, duration rule, and the auxiliary

harnesses/codex.toml

@@ -34,17 +34,24 @@ unresolved_phrase = "If it does not load as a Codex skill"
 header = "## Skills\n"
 item = "\n- {name}: {description} (file: {path})"
 
-# Codex's JSONL exposes no deterministic skill-tool event, so the
-# `__skill_invoked` meta-check uses the LLM-judge fallback.
+# Codex's JSONL exposes no deterministic skill-tool event. It does preserve a
+# completed command's literal arguments and exit code, so the meta-check uses
+# an exact staged-SKILL.md access signature instead.
 #
 # The JSONL stream is flat (every tool event self-contained), so its summary is
 # declarative. Permission denials are independent: the named reader correlates
 # the sibling stderr capture where Codex reports structural router rejections.
 [transcript]
 events_filename = "codex-events.jsonl"
-surfaces_skill_invocation = false
+surfaces_skill_invocation = true
 permission_denials_parser = "codex-items"
 
+[transcript.skill_access]
+tool = "command_execution"
+command_arg = "command"
+exit_code_arg = "exit_code"
+read_commands = ["cat", "head", "sed", "tail"]
+
 [transcript.extract.tools]
 where = { type = "item.completed" }
 item = "item"

harnesses/template.toml

@@ -124,18 +124,28 @@ label = "{label}"
 ## meanwhile.
 ## VERIFY: capture a real events file and check its line shapes against the reader you declare —
 ## an unknown parser name fails the schema gate listing the allowed values. Leave
-## surfaces_skill_invocation false unless you verified the stream yields a deterministic
-## skill-invocation event (false routes the __skill_invoked meta-check to the LLM judge). When
-## true, skill_tool / skill_arg name the skill tool and the argument carrying the skill
-## identifier (defaults: "Skill" / "skill" — Claude Code's spellings; OpenCode uses
-## "skill" / "name").
+## Leave surfaces_skill_invocation false unless the stream yields deterministic native invocation
+## or staged-file access evidence (false routes __skill_invoked to a clearly labeled behavioral-
+## influence fallback). For a native event, skill_tool / skill_arg name the tool and identifier
+## argument (defaults: "Skill" / "skill"; OpenCode uses "skill" / "name"). A harness that exposes
+## only successful shell reads may instead declare [transcript.skill_access]: tool is a [tools]
+## shell name, command_arg and exit_code_arg locate its fields, and read_commands lists literal
+## executable basenames. The exact staged SKILL.md path must be one literal argument.
 # [transcript]
 # events_filename = "{label}-events.jsonl"
 # parser = "codex-items"
 # permission_denials_parser = "codex-items"
-# surfaces_skill_invocation = false
-# skill_tool = "Skill"
-# skill_arg = "skill"
+# surfaces_skill_invocation = true
+#
+# [transcript.skill_access]
+# tool = "command_execution"
+# command_arg = "command"
+# exit_code_arg = "exit_code"
+# read_commands = ["cat", "sed"]
+#
+## Native-tool alternative to [transcript.skill_access]:
+##   skill_tool = "Skill"
+##   skill_arg = "skill"
 #
 # [transcript.extract.session_surface]
 # where = { type = "system", subtype = "init" }

schema/harness-descriptor.schema.json

@@ -244,7 +244,7 @@
         },
         "surfaces_skill_invocation": {
           "type": "boolean",
-          "description": "Whether the transcript deterministically surfaces a skill invocation (false routes the meta-check to the LLM-judge fallback). For extract descriptors, leave false unless the mapping verifiably yields invocations named `Skill` carrying an `args.skill` field."
+          "description": "Whether the transcript deterministically surfaces native skill invocation or exact staged-file access (false routes the meta-check to a behavioral-influence fallback)."
         },
         "skill_tool": {
           "type": "string",
@@ -255,6 +255,24 @@
           "type": "string",
           "minLength": 1,
           "description": "Argument of the skill-invocation tool that carries the staged slug (default \"skill\"; OpenCode declares \"name\")."
+        },
+        "skill_access": {
+          "type": "object",
+          "required": ["tool", "command_arg", "exit_code_arg", "read_commands"],
+          "additionalProperties": false,
+          "description": "Deterministic exact-path access signature for a harness without a native skill tool. The named tool must complete successfully and its command argument must contain the task's exact staged SKILL.md path as a literal shell argument.",
+          "properties": {
+            "tool": { "type": "string", "minLength": 1 },
+            "command_arg": { "type": "string", "minLength": 1 },
+            "exit_code_arg": { "type": "string", "minLength": 1 },
+            "read_commands": {
+              "type": "array",
+              "minItems": 1,
+              "uniqueItems": true,
+              "items": { "type": "string", "minLength": 1 },
+              "description": "Literal executable basenames whose successful invocation with the exact staged path counts as a content read."
+            }
+          }
         }
       }
     },

schema/run-record.schema.json

@@ -25,7 +25,11 @@
     },
     "skill_path": {
       "type": ["string", "null"],
-      "description": "Absolute path to the SKILL.md the subagent could load, or null if no skill was provided (without_skill condition)."
+      "description": "Absolute path to the campaign-owned SKILL.md input, or null if no skill was provided. The task-specific staged location is staged_skill_path."
+    },
+    "staged_skill_path": {
+      "type": "string",
+      "description": "Exact task-environment path to the staged scalar SKILL.md. Absent for control, unstaged, and historical runs."
     },
     "skills": {
       "type": "array",
@@ -107,7 +111,11 @@
       "properties": {
         "name": { "type": "string" },
         "skill_path": { "type": "string" },
-        "staged_skill_slug": { "type": ["string", "null"] }
+        "staged_skill_slug": { "type": ["string", "null"] },
+        "staged_skill_path": {
+          "type": "string",
+          "description": "Exact task-environment path to this member's staged SKILL.md. Absent before a task env is selected and in historical records."
+        }
       }
     },
     "responderOutcome": {

src/adapters/descriptor/transcript_section.rs

@@ -9,6 +9,16 @@ use serde::{Deserialize, Serialize};
 
 use super::{default_true, is_true};
 
+/// A deterministic staged-skill access encoded in a successful native command
+/// item rather than a dedicated skill tool event.
+#[derive(Debug, Clone, Deserialize, Serialize)]
+pub struct SkillAccessSection {
+    pub tool: String,
+    pub command_arg: String,
+    pub exit_code_arg: String,
+    pub read_commands: Vec<String>,
+}
+
 #[derive(Debug, Clone, Deserialize, Serialize)]
 pub struct TranscriptSection {
     pub events_filename: String,
@@ -27,15 +37,19 @@ pub struct TranscriptSection {
     pub extract: Option<ExtractSpec>,
     #[serde(default = "default_true", skip_serializing_if = "is_true")]
     pub surfaces_skill_invocation: bool,
-    /// Tool name of the deterministic skill-invocation event the
-    /// `__skill_invoked` meta-check matches (default `"Skill"` — Claude
-    /// Code's Skill tool).
+    /// Tool name of a deterministic native skill-invocation event (default
+    /// `"Skill"` — Claude Code's Skill tool). Mutually exclusive with
+    /// [`Self::skill_access`].
     #[serde(skip_serializing_if = "Option::is_none")]
     pub skill_tool: Option<String>,
     /// Argument of the skill-invocation tool that carries the staged slug
     /// (default `"skill"`).
     #[serde(skip_serializing_if = "Option::is_none")]
     pub skill_arg: Option<String>,
+    /// Successful shell-command signature whose literal argument must equal
+    /// the task's exact staged `SKILL.md` path.
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub skill_access: Option<SkillAccessSection>,
 }
 
 impl TranscriptSection {

src/adapters/descriptor/validation.rs

@@ -37,6 +37,7 @@ const CHECKS: &[Check] = &[
     check_guard_engine_fields,
     check_guard_verdict_template,
     transcript::check_tool_vocabulary,
+    transcript::check_skill_evidence,
     transcript::check_tiers,
     conversation::validate,
     check_tool_roles_disjoint,

src/adapters/descriptor/validation/tests/transcript.rs

@@ -112,6 +112,72 @@ fn accepts_an_independent_permission_denials_parser_with_extract_ingest() {
     load_descriptor(&descriptor, "test.toml").expect("denial parsing should compose");
 }
 
+#[test]
+fn accepts_a_successful_exact_path_skill_access_signature() {
+    let descriptor = format!(
+        "{TOOLED}\n[transcript]\nevents_filename = \"demo-events.jsonl\"\n\
+         parser = \"codex-items\"\nsurfaces_skill_invocation = true\n\n\
+         [transcript.skill_access]\n\
+         tool = \"command_execution\"\n\
+         command_arg = \"command\"\n\
+         exit_code_arg = \"exit_code\"\n\
+         read_commands = [\"cat\", \"sed\"]\n"
+    );
+
+    load_descriptor(&descriptor, "test.toml")
+        .expect("an exact-path access signature should be descriptor-declared");
+}
+
+#[test]
+fn rejects_skill_access_that_is_not_a_true_exclusive_shell_signal() {
+    for (extra_transcript, expected) in [
+        (
+            "surfaces_skill_invocation = false\n",
+            "surfaces_skill_invocation",
+        ),
+        (
+            "surfaces_skill_invocation = true\nskill_tool = \"Skill\"\n",
+            "skill_tool",
+        ),
+    ] {
+        let err = err_of(&format!(
+            "{TOOLED}\n[transcript]\nevents_filename = \"demo-events.jsonl\"\n\
+             parser = \"codex-items\"\n{extra_transcript}\n\
+             [transcript.skill_access]\n\
+             tool = \"command_execution\"\n\
+             command_arg = \"command\"\n\
+             exit_code_arg = \"exit_code\"\n\
+             read_commands = [\"cat\", \"sed\"]\n"
+        ));
+        assert!(err.contains(expected), "{err}");
+    }
+
+    let err = err_of(&format!(
+        "{TOOLED}\n[transcript]\nevents_filename = \"demo-events.jsonl\"\n\
+         parser = \"codex-items\"\nsurfaces_skill_invocation = true\n\n\
+         [transcript.skill_access]\n\
+         tool = \"file_change\"\n\
+         command_arg = \"command\"\n\
+         exit_code_arg = \"exit_code\"\n\
+         read_commands = [\"cat\", \"sed\"]\n"
+    ));
+    assert!(err.contains("tools.shell"), "{err}");
+}
+
+#[test]
+fn rejects_skill_access_read_commands_that_are_not_literal_basenames() {
+    let err = err_of(&format!(
+        "{TOOLED}\n[transcript]\nevents_filename = \"demo-events.jsonl\"\n\
+         parser = \"codex-items\"\nsurfaces_skill_invocation = true\n\n\
+         [transcript.skill_access]\n\
+         tool = \"command_execution\"\n\
+         command_arg = \"command\"\n\
+         exit_code_arg = \"exit_code\"\n\
+         read_commands = [\"/bin/cat\"]\n"
+    ));
+    assert!(err.contains("literal executable basenames"), "{err}");
+}
+
 #[test]
 fn rejects_transcript_with_neither_parser_nor_extract() {
     for auxiliary in ["", "permission_denials_parser = \"codex-items\"\n"] {

src/adapters/descriptor/validation/transcript.rs

@@ -13,6 +13,60 @@ pub(super) fn check_tool_vocabulary(d: &HarnessDescriptor) -> Result<(), String>
     Ok(())
 }
 
+/// Exact-path access is an alternative to a native skill-tool signature. Its
+/// tool must be declared as shell so the descriptor cannot assign command-line
+/// semantics to an unrelated transcript item.
+pub(super) fn check_skill_evidence(d: &HarnessDescriptor) -> Result<(), String> {
+    let Some(transcript) = &d.transcript else {
+        return Ok(());
+    };
+    let Some(access) = &transcript.skill_access else {
+        return Ok(());
+    };
+    if !transcript.surfaces_skill_invocation {
+        return Err(
+            "transcript.skill_access requires surfaces_skill_invocation = true; otherwise the \
+             descriptor declares deterministic evidence and disables it at the same time"
+                .into(),
+        );
+    }
+    if transcript.skill_tool.is_some() || transcript.skill_arg.is_some() {
+        return Err(
+            "transcript.skill_access cannot be combined with skill_tool or skill_arg; declare \
+             either exact-path access or a native skill-tool signature"
+                .into(),
+        );
+    }
+    if !d.tools.shell.contains(&access.tool) {
+        return Err(format!(
+            "transcript.skill_access.tool {:?} is not declared in tools.shell; exact-path \
+             command evidence must use a shell tool",
+            access.tool
+        ));
+    }
+    if access.command_arg == access.exit_code_arg {
+        return Err(
+            "transcript.skill_access command_arg and exit_code_arg must name distinct fields"
+                .into(),
+        );
+    }
+    if access.read_commands.iter().any(|command| {
+        command.is_empty()
+            || command == "."
+            || command == ".."
+            || !command
+                .chars()
+                .all(|ch| ch.is_ascii_alphanumeric() || matches!(ch, '-' | '_' | '+' | '.'))
+    }) {
+        return Err(
+            "transcript.skill_access.read_commands must contain literal executable basenames; \
+             paths, whitespace, and shell syntax are not allowed"
+                .into(),
+        );
+    }
+    Ok(())
+}
+
 /// Transcript ingest has one primary summary reader: a named code parser or
 /// declarative summary outputs. Independent denial and session-surface readers
 /// may compose around it.