Skip to content

fix(codex): verify staged skill access - #315

Merged
slowdini merged 1 commit into
devfrom
fix/codex-skill-access-evidence
Sep 3, 2026
Merged

fix(codex): verify staged skill access#315
slowdini merged 1 commit into
devfrom
fix/codex-skill-access-evidence

Conversation

@slowdini

@slowdini slowdini commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Summary

Closes #309.

Codex JSONL has no dedicated skill-invocation event, but a bounded Codex CLI 0.152.1 probe showed that completed command_execution items preserve the command text and numeric exit code. This change uses that stable event shape as deterministic skill-access evidence.

  • Persist each task's exact staged SKILL.md path in dispatch and run artifacts.
  • Add a descriptor-declared transcript.skill_access signature with the shell tool, command field, exit-code field, and permitted reader basenames.
  • Grade __skill_invoked locally with confidence: 1.0 when a successful reader command contains the exact staged path as a literal argument.
  • Keep the LLM path only when a run has no usable deterministic signature, and label it as behavioral-influence evidence rather than invocation proof.

Evidence boundary

The deterministic check is deliberately narrow:

  • The command must use a descriptor-approved reader (cat, head, sed, or tail for Codex).
  • The task-specific staged path must be a complete literal shell word.
  • The native command must report exit code zero.
  • Dynamic, partial, live-source, and other treatment-member paths do not count.
  • Final-message wording and skill-name mentions do not count.
  • Compound and piped commands do not count because one aggregate exit code cannot prove which command succeeded.

Multi-skill grading keeps conditions.json as the authoritative member roster and uses matching run.json entries only for task-specific staged paths, so an incomplete copied roster cannot suppress a member's check.

Compatibility and artifacts

  • Claude Code, Cline, and OpenCode retain their native skill-tool signatures.
  • Scalar treatments retain __skill_invoked.json; multi-skill treatments retain indexed __skill_invoked__skill-N.json files.
  • staged_skill_path is additive and optional in run-record schemas, preserving historical artifact compatibility.
  • Descriptors without a trustworthy signal retain one behavioral-influence fallback task per treatment member.
  • The harness descriptor schema and scaffold document the mutually exclusive native-tool and exact-path signature forms.

Before this change, Codex always asked an LLM to infer invocation from behavioral fingerprints. After this change, staged Codex runs produce local transcript-check results from exact-path access, and persuasive response text wording cannot turn missing access into a pass.

Documentation and fixture

  • Update the shipped judging guide and descriptor scaffold.
  • Record the Codex CLI 0.152.1 probe and event contract in the implementation notes.
  • Add a minimal synthetic JSONL fixture based on the observed completed command shape.

Verification

  • cargo fmt --check
  • cargo clippy --all-targets -- -D warnings
  • cargo test --quiet — 972 library, 220 CLI, and 211 run tests passed (1,403 total)
  • cargo build
  • Pre-commit formatting and clippy hooks passed.
  • Pre-push full test hook passed.

Persist each task's staged SKILL.md path and require a successful transcript read command against that exact path. Preserve native skill-tool checks and label no-signal judging as behavioral influence rather than invocation proof.\n\nCloses #309
@slowdini
slowdini merged commit 82d657d into dev Sep 3, 2026
7 checks passed
@slowdini
slowdini deleted the fix/codex-skill-access-evidence branch September 3, 2026 06:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use deterministic Codex skill-access evidence for __skill_invoked

1 participant