Merge pull request #160 from slowdini/dev

Release v0.5.1

Author: slowdini

.github/PULL_REQUEST_TEMPLATE/harness-descriptor.md

@@ -19,7 +19,8 @@ Title: feat(harness): add <label> descriptor
 - [ ] This PR adds/changes descriptor **data only** — no parser, slug, shadow,
       or guard code (those are separate one-capability-per-PR contributions)
 - [ ] `[transcript]`/`[shadow]`/`[staging]` (if declared) reuse an existing named
-      capability (`claude-stream-json` / `codex-items` / `opencode` / `claude-plugins`)
+      capability (`claude-stream-json` / `codex-items` / `opencode-events` / `opencode` /
+      `claude-plugins` / `codex-skills` / `opencode-skills`)
       or ingest through a `[transcript.extract]` block (declarative data)
 - [ ] No `[guard]` table and no `run.supports_guard = true` in this PR
 

AGENTS.md

@@ -4,7 +4,8 @@
 
 `eval-magic` is a Rust CLI crate. The binary entry point is `src/main.rs`; reusable logic lives in
 `src/lib.rs` and submodules such as `cli/`, `pipeline/`, `sandbox/`, `validation/`, and
-`workspace/`. JSON schemas are tracked in `schema/`, harness descriptors in `harnesses/`, shared
+`workspace/`. JSON schemas are tracked in `schema/`, harness descriptors (plus embedded harness
+assets such as the OpenCode write-guard plugin template) in `harnesses/`, shared
 prompt profiles in `profiles/`, and development docs (the harness enhancement contract,
 per-harness notes) in `docs/`. Integration
 tests are split by surface area under `tests/cli/` and `tests/run/`; unit tests usually live

Cargo.lock

@@ -1,6 +1,6 @@
 [package]
 name = "eval-magic"
-version = "0.5.0"
+version = "0.5.1"
 edition = "2024"
 description = "One-stop CLI for running skill evals — measure whether an agent skill actually shifts behavior."
 license = "MIT"

Cargo.toml

@@ -4,7 +4,7 @@
 
 An eval dispatches a fresh subagent twice per test case — once with the skill loaded, once without (or old version vs. new) — and grades both outputs against assertions. The pass-rate delta tells you whether the skill is worth shipping or the change is worth landing. The runner builds the workspace, stages skills for discovery, generates dispatch prompts, assembles run records from transcripts, grades, and aggregates; your agent harness supplies the one thing the runner never does itself: dispatching the subagents.
 
-`eval-magic` ships as a dependency-less prebuilt binary under the command name `eval-magic`. Every artifact follows a documented JSON Schema, so records grade the same way regardless of where they were authored. **Claude Code and Codex CLI are fully wired harnesses today**; OpenCode has native staging support; see [Harnesses](#harnesses) for per-harness enhancement support. From inside an agent session, running an eval is as simple as: *"Install eval-magic and help me run an eval on my-skill."*
+`eval-magic` ships as a dependency-less prebuilt binary under the command name `eval-magic`. Every artifact follows a documented JSON Schema, so records grade the same way regardless of where they were authored. **Claude Code and Codex CLI are fully wired harnesses today**; OpenCode has native staging and transcript-ingest support; see [Harnesses](#harnesses) for per-harness enhancement support. From inside an agent session, running an eval is as simple as: *"Install eval-magic and help me run an eval on my-skill."*
 
 This README is the complete operating guide: install, author cases, run the loop, read results, and keep a baseline. For the full flag-by-flag reference, run `eval-magic --help` (and `eval-magic <subcommand> --help`). For *when and why* to write an eval at all — the methodology, the decision to test, designing cases under pressure — see the [`slow-powers`](https://github.com/slowdini/slow-powers) plugin's `evaluating-skills` skill, which owns that craft.
 
@@ -250,7 +250,7 @@ A subagent that runs an eval should start in an environment that mirrors a real
 
 For the `without_skill` / baseline condition, the dispatch reflects "this skill is unavailable, others remain" when siblings were opted in with `--skill-dir`; otherwise it measures the skill against a clean no-skill baseline. `--bootstrap` is separate from parity: it injects product-specific framing inside the `<session-start-context>` block and does not enumerate skills.
 
-**Parity is only as clean as your session.** Staging controls what the runner *adds*, not what your session already *loaded*. Subagents dispatched in-process share the parent session's plugins, so an installed plugin exposing a same-named skill is still discoverable and contaminates both arms — the staging slug stops an on-disk collision, not runtime discovery. The runner can't unload a live plugin; on Claude Code it emits a build-time *plugin-shadow* warning (also surfaced in `benchmark.json`'s `validity_warnings`) that lists the per-dispatch isolation options inline. Closing it is a launch-time step for whoever dispatches.
+**Parity is only as clean as each dispatch's live environment.** Staging controls what the runner *adds*, not the user, repository, admin, or plugin skills the harness's one-shot CLI also discovers. A live copy of the logical eval skill can therefore contaminate both arms — the staging slug stops an on-disk collision, not runtime discovery. The runner can't unload a live skill; Claude Code, Codex, and OpenCode emit a build-time *skill-shadow* warning (recorded in the legacy `plugin-shadow.json` artifact and surfaced in `benchmark.json`'s `validity_warnings`) with harness-specific isolation guidance. Closing it is a launch-time step for whoever dispatches.
 
 ## Harnesses
 
@@ -266,15 +266,13 @@ Every eval test and judge is dispatched the same way: through the harness's one-
 
 This table is the source of truth for per-harness enhancement support:
 
-| Harness | Native staging | Dispatch recipes | Transcript ingest | Model flag | Write guard |
-|---------|:--------------:|:----------------:|:-----------------:|:----------:|:-----------:|
-| **Claude Code** | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Codex** | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **OpenCode** | ✅ | ❌¹ | ❌ | ❌ | ❌ |
+| Harness | Native staging | Dispatch recipes | Transcript ingest | Model flag | Write guard | Shadow preflight |
+|---------|:--------------:|:----------------:|:-----------------:|:----------:|:-----------:|:----------------:|
+| **Claude Code** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **Codex** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **OpenCode** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
 
-¹ `run --harness opencode` stages skills and emits native dispatch prompts, but prints manual `opencode run` guidance instead of a copy-pasteable recipe.
-
-A missing enhancement degrades fidelity, never correctness — every column has a fallback, and the `run` preflight warns naming it: without native staging, `--no-stage` inlines each `SKILL.md` into its dispatch prompt; without transcript ingest, `transcript_check` assertions grade as unverifiable and `llm_judge` carries the grading (tokens and duration go unrecorded); without a model flag, `--agent-model` / `--judge-model` are recorded as provenance only; without a write guard, the run continues unguarded (auto-arm quietly stays off; an explicit `--guard` warns) and `detect-stray-writes` audits after the fact; without dispatch recipes, `RUNBOOK.md` / `dispatch-manifest.md` carry handoff guidance without a copy-pasteable per-task command. Supported enhancements are provided automatically — the write guard arms on every staged run of a guard-capable harness unless `--no-guard` opts out.
+A missing enhancement degrades fidelity, never correctness — every column has a fallback: without native staging, `--no-stage` inlines each `SKILL.md` into its dispatch prompt; without transcript ingest, `transcript_check` assertions grade as unverifiable and `llm_judge` carries the grading (tokens and duration go unrecorded); without a model flag, `--agent-model` / `--judge-model` are recorded as provenance only; without a write guard, the run continues unguarded (auto-arm quietly stays off; an explicit `--guard` warns) and `detect-stray-writes` audits after the fact; without shadow preflight, no automatic live-skill collision scan runs; without dispatch recipes, `RUNBOOK.md` / `dispatch-manifest.md` carry handoff guidance without a copy-pasteable per-task command. Supported enhancements are provided automatically — the write guard arms on every staged run of a guard-capable harness unless `--no-guard` opts out, and the `run` preflight names actionable fallbacks where it can.
 
 Per-harness implementation notes for developers wiring features live in [docs/claude-notes.md](docs/claude-notes.md), [docs/codex-notes.md](docs/codex-notes.md), and [docs/opencode-notes.md](docs/opencode-notes.md).
 

README.md

@@ -135,8 +135,11 @@ you get the full feature from configuration alone:
 
 - `transcript.parser = "claude-stream-json"` — Claude Code `-p --output-format stream-json` events.
 - `transcript.parser = "codex-items"` — Codex `item.started`/`item.completed` JSONL.
+- `transcript.parser = "opencode-events"` — OpenCode `run --format json` `tool_use`/`text`/`step_finish` events.
 - `staging.slug_capability = "opencode"` — OpenCode's sanitizing slug rules.
 - `shadow.preflight = "claude-plugins"` — the Claude plugin/global-skills shadow scan.
+- `shadow.preflight = "codex-skills"` — the Codex repo/user/admin/plugin skill scan.
+- `shadow.preflight = "opencode-skills"` — the OpenCode project/global `.opencode`/`.claude`/`.agents` skill scan.
 
 For example, a harness that logs Codex-compatible item JSONL gets full transcript ingest — parsed
 tool invocations, `transcript_check` grading, the works — with:
@@ -211,8 +214,10 @@ Dotted paths (`"usage.input_tokens"`, `"item.text"`) descend nested objects only
 array indexing, and keys containing literal dots are unaddressable. Malformed JSONL lines are
 silently skipped, as with every parser. `[transcript]` requires the `[tools]` write/shell
 vocabulary alongside it (the stray-writes audit classifies by it). Leave
-`surfaces_skill_invocation` false unless the mapping verifiably yields invocations named `Skill`
-carrying an `args.skill` field.
+`surfaces_skill_invocation` false unless the mapping verifiably yields a deterministic
+skill-invocation event; when true, the `__skill_invoked` meta-check matches the tool named by
+`skill_tool` whose `skill_arg` argument equals the staged slug (defaults `"Skill"` / `"skill"` —
+Claude Code's spellings; OpenCode declares `"skill"` / `"name"`).
 
 **If a stream needs more than these primitives, it's a code capability, not a bigger DSL.** The
 line is cross-event state: a stream whose tool results arrive in *separate* records joined by id
@@ -267,7 +272,8 @@ beyond a mechanical registration. What counts as data vs code:
   `config_dirs`, the `[dispatch]` templates, `[model]`, `[staging]` + `[skills_block]`,
   `[tools]`, the `[run]` booleans, a `[transcript.extract]` block (the declarative tier is pure
   data), and `[transcript]` / `[shadow]` **when they reuse an existing named capability**
-  (`claude-stream-json`, `codex-items`, `opencode`, `claude-plugins`).
+  (`claude-stream-json`, `codex-items`, `opencode-events`, `opencode`, `claude-plugins`,
+  `codex-skills`, `opencode-skills`).
 - **Code — one capability per PR, separate from the descriptor PR:** a new transcript parser,
   slug capability, or shadow preflight (each is `src/adapters/capabilities.rs` + a
   `src/adapters/<harness>/` module + a schema enum entry), and guard support (`[guard]` data is

docs/byoh.md

@@ -14,6 +14,7 @@ references:
 |------|--------------|
 | `harnesses/codex.toml` | the descriptor — every declarative value + capability references |
 | `transcript.rs` | `item.completed` event-stream parsing (`codex-items`) |
+| `skill_shadow.rs` | repo/user/admin/plugin skill collision scan (`codex-skills`) + reporting |
 
 The write guard has no per-harness code: the descriptor's `[guard]` block (hook file, matcher,
 hook-entry and `{"decision": "block"}` verdict templates) is rendered by the generic engine in
@@ -46,6 +47,37 @@ so the staged skill-under-test's frontmatter is rewritten to the eval slug
 (`rewrites_frontmatter_name` true) and the available-skills block advertises the slug
 (`advertises_staged_slug_name` true).
 
+Codex can also inject `--bootstrap` content into a no-stage dispatch: the bootstrap remains in
+`<session-start-context>`, while the skill-under-test is inlined separately for the treatment arm.
+`--stage-name` still requires staging because it names an on-disk staged copy.
+
+## Isolating from live skills and plugins
+
+Every `codex exec` can discover skills beyond the eval env. Before dispatch, the `codex-skills`
+preflight compares each logical eval skill name with:
+
+- `.agents/skills` at each repository ancestor of the dispatch cwd;
+- `$HOME/.agents/skills`;
+- `/etc/codex/skills`; and
+- skills in enabled installed plugins reported by `codex plugin list --json`, under
+  `$CODEX_HOME/plugins/cache/<marketplace>/<plugin>/<version>/skills`.
+
+Direct skill directories are matched by the `name:` in `SKILL.md` frontmatter, not the folder
+name. Missing directories, malformed skills, and unavailable or invalid plugin-list output are
+ignored; a plugin-list failure does not suppress findings from the direct directories. Findings
+produce a build-time Codex banner and the backward-compatible `plugin-shadow.json` artifact;
+`aggregate` turns the same report into Codex-specific `benchmark.json` validity warnings.
+
+eval-magic detects but cannot unload these sources. Disable a conflicting installed plugin from
+Codex's `/plugins` UI, or move/rename a conflicting repo, user, or admin skill before dispatch.
+For a user skill only, a clean `HOME` can isolate `$HOME/.agents/skills`; preserve `CODEX_HOME` if
+the dispatch still needs the existing Codex configuration. That does not isolate plugins stored
+under `CODEX_HOME` or repository/admin skills.
+
+**Known limit:** Codex also ships bundled system skills, but currently exposes no stable
+enumeration mechanism for them. The preflight therefore cannot detect a collision with a bundled
+system skill; verify that case manually when relevant.
+
 ## Transcript (`item.completed`)
 
 `item.completed` events whose item type is not an agent message / reasoning / plan update become